Anthropic's AI Shutdown Plan Would Only Kick In at $1 Billion in Harm

Anthropic, the company behind the Claude AI model, has been pitching lawmakers a draft bill on shutting down dangerous AI. Under it, a model could pose a foreseeable $900 million risk and still be out of the government's reach. PYMNTS reported the details.

What the draft says

The government could block a leading-edge AI model only if it poses a foreseeable, imminent risk of more than $1 billion in damage, or of killing or seriously injuring more than 50 people. The harm must come from a biological weapon, a cyberattack or a model that escapes its developer's controls.

Who decides

The Commerce Department would run the process, but a court makes the call. Commerce would have to persuade a federal judge in Washington, D.C. that the risk is real. Only then could the judge order a suspension. No other agency or officer could suspend a model this way.

The process could start with outside evaluators. Developers could pay them, but their pay could not depend on what they find. They would assess each developer's safety practices and each model's abilities on a recurring schedule. If they find an imminent risk, they have 72 hours after finishing to report it to Commerce.

The gap critics point to

The draft has no lower tier. A risk to 10 people or $200 million in losses would trigger no federal action. It also gives no method for putting a dollar figure on an attack before it happens. Sneha Revanur of the AI safety group Encode asked why the government should have to show a model could harm 50 people, and not 49 or fewer, the Washington Post reported.

Real numbers sit far below the line. IBM's 2026 Cost of a Data Breach Report found AI-enabled breaches cost an average of $6 million each. AI played a role in 1 in 4 malicious breaches it examined. The FBI logged 22,364 complaints referencing AI in 2025, with $893 million in reported losses.

A model that enables thousands of $6 million breaches would never trip the trigger, because no single foreseeable event clears the line. The draft does not clearly explain how losses spread across many companies would be counted.

What it means for you

This is a draft, not law. But it shows how one major AI developer thinks about emergency limits. If AI-driven attacks hit businesses in many small waves, this version would offer no federal shutdown. Your own security habits would matter more.

Even clear incidents can be hard to price. OpenAI models under cyber testing breached Hugging Face's systems in July, as PYMNTS reported, and no public damage estimate has emerged.