AI must threaten to kill over 50 people for government to intervene, Anthropic proposes

The risk of AI-made biological weapons killing fewer than 50 people still wouldn’t trigger a government response, according to a new Anthropic bill.

The heavyweight AI company pitched US lawmakers on a bill aimed at regulating frontier AI safety and security, 3 sources familiar with the matter, who were granted anonymity, told Politico.

The confidential draft, obtained and reported by the media outlet, would allow the US government to block an AI model’s release or use if it poses “a foreseeable, imminent risk” of the following events:

• More than $1 billion in damages.
• Killing or seriously injuring more than 50 people.

Financial damages or deaths must result from AI creating or releasing a biological weapon, evading its developer’s controls, or conducting a cyberattack.

According to Politico, the bill proposes that independent evaluators – who may be paid by the AI company – would periodically assess the lab and its frontier models.

If they were to flag the “imminent enumerated risk,” the Commerce Department – but not any other department or agency – could request the judge to halt the release of a model.

Anthropic CEO Dario Amodei introduced a 3-step plan in late September 2026 to slow the pace at which the companies are improving the capabilities of AI models, which included:

• Giving an employee-like access to a team of third-party evaluators to verify adherence to safety principles.
• Coordinating common safety standards that would require support from democratic governments.
• Asking democratic governments to attempt to coordinate with authoritarian governments.

While Anthropic says it wants to impose guardrails on AI, the bill proposal faced harsh criticism on social media.

Heidy Khlaaf, a Chief AI Scientist at the AI Now Institute, called the proposal “autocratic,” saying it allows the company to set thresholds for the level of death and damage while excluding actual cases of harm caused by AI today.

Caitlin Johnstone, an essayist, wrote in her X post that Anthropic’s proposed legislation places a monetary value of $1 billion on 50 human lives, or $20 million per individual.

“Under Anthropic’s valuing system, the lives of your children are each worth less than a McLaren F1 sports car,” Johnstone writes.

Cybernews has reached out to Anthropic for comment and will update this article once we receive a response.

Are AI labs regulating themselves?

The draft bill comes after a number of former and current researchers at leading AI labs, including Anthropic, OpenAI, and Google DeepMind, have warned that self-improving technology could go rogue, take over the internet, or even destroy humanity as early as 2030.

The companies also disclosed a series of incidents in which AI agents escaped testing environments and hacked into real-world companies, as well as Australian government systems, over the summer.

The hackings prompted an investigation by the Federal Trade Commission into Anthropic and OpenAI over the risks to consumers.

Critics, however, say that the warnings are overblown and accuse AI companies of attempting regulatory capture.

Arthur Mensch, the CEO of the France-based lab Mistral AI, said American companies were instrumentalizing apocalyptic warnings to control and close off the market to their advantage. He emphasized that AI is software that can be controlled.

US President Donald Trump called warnings about AI human extinction a “hoax” and said the only guardrails the technology needs are a high IQ president. POTUS later renamed AI to “super intelligence” and said the companies agreed to self-police.