OpenAI’s ex-insider: ceding to Anthropic’s guardrails would pose problems to Pentagon

A former OpenAI employee says that Anthropic’s openness to striking a deal with the Pentagon after it was blacklisted suggests that artificial intelligence (AI) companies have volatile policies regarding the technology’s military use, which may harm civilians in war zones.

After a week of the Pentagon’s threats, Anthropic refused to drop its guardrails, stating that the company objects to its tools being used for “mass domestic surveillance” and “fully autonomous weapons.”

As a result, the Department of War (DoW) declared Anthropic a “supply chain risk,” and President Donald Trump ordered the federal government to immediately stop using the company's technology.

However, the US military reportedly used Anthropic’s Claude during its attack on Iran over the weekend, just hours after blacklisting it, according to the Wall Street Journal.

While many praised Anthropic for a principled position – although it said it was open to a new deal with the Pentagon – its major competitor, OpenAI, announced an agreement with the DoW.

The company claims the new deal has more guardrails than previous agreements, with its red lines including using its AI tools for mass domestic surveillance, directing autonomous weapons systems, and using its high-stakes automated decisions.

Pentagon may be asserting authority

Sarah Shoker, a former geopolitics team lead at OpenAI who consulted the company on military AI uses, wrote in her Substack post that US policy, specifically the “Project Maven” memo, emphasizes that “appropriate levels of human judgement” are applied to semi- and fully-autonomous weapons systems.

This suggests that Anthropic and the Pentagon essentially agree on the military uses of AI. However, the Pentagon may interpret Anthropic’s “no” as an affront to the DoW’s authority.

Therefore, agreeing to Anthropic’s guardrails may create greater problems for the DoW in the event of a future disagreement.

There may also be miscommunication, as “people are speaking past one another,” she writes.

Shoker adds there could be a disagreement over what it means to apply “appropriate levels of human judgement” to an autonomous weapons system, especially given that there’s no consensus on what “human in the loop” means.

“Militaries are still trying to develop new testing and evaluation procedures for reducing problems such as over-reliance in human-AI teams. It’s possible that Anthropic disagreed with how ‘human supervision’ (broadly speaking) would be put into practice,” she writes.

In an interview with CBS News, Anthropic’s CEO Dario Amodei emphasized that the company’s red lines don’t extend to the “partially autonomous weapons” that are used in Ukraine.

He didn’t rule out that adversaries could have fully autonomous weapons in the future. In this case, such weapons would be “needed for the defense of democracy.”

However, Amodei emphasized that AI systems nowadays aren’t reliable enough to make fully autonomous weapons, as they can be unpredictable.

“And there's an oversight question, too. If you have a large army of drones or robots that can operate without any human oversight, where there aren’t human soldiers to make the decisions about who to target, who to shoot at, that – that presents concerns,” he said.

According to Shoker, this means that the commitments made today by AI companies may change tomorrow “based on criteria that are not self-evident.”

“Can we please agree that democracy can’t survive on the goodwill of CEOs who have 200 million dollars to burn? This is not a sustainable solution,” she adds.

Anil Dash, an entrepreneur and writer, noted in his Substack post that praise for Anthropic's refusal of the Pentagon’s demands shouldn’t be “overly effusive.”​

“The bar cannot be set so impossibly low that we celebrate merely refusing to directly, intentionally enable war crimes like the repeated bombing of unknown targets in international waters, in direct violation of both US and international law,” he wrote.

The US military faces war crime allegations after it allegedly disguised its aircraft as a civilian plane to carry out an attack on a suspected drug smuggling boat in the southern Caribbean Sea in September 2025, according to the New York Times reports.

The bar cannot be set so impossibly low that we celebrate merely refusing to directly, intentionally enable war crimes like the repeated bombing of unknown targets in international waters, in direct violation of both US and international law.

Anil Dash

Civilians lose from volatile AI military use policies

Amodei told CBS News that Anthropic is still trying to reach a deal with the DoW and is interested in working with them “as long as it is in line with our red lines.”

At the same time, he urged the Congress to “catch up with where the technology is going.”

Shoker points out that company policies around military AI use are volatile and are likely to change again. Moreover, they are subject to a variety of invisible pressures, including from the public.

Meanwhile, the biggest losers of such instability are everyday people and civilians in conflict zones, as large language models (LLMs) are black boxes.

She writes, “Defense and intelligence agencies are (mostly) black boxes as a matter of policy. Conflict zones are black boxes due to the fog of war and disinformation.”

Unlock more exclusive Cybernews content on YouTube.