Anthropic has privately pitched US lawmakers a draft artificial intelligence safety bill that would allow a court to halt an advanced AI model when it poses a foreseeable and imminent risk of killing or seriously injuring more than 50 people, or causing more than $1 billion in property damage. The proposal seeks to establish a federal framework for regulating powerful AI systems, including when their release or use could be stopped over catastrophic safety concerns.
The draft, reported by Politico in October 2026 and subsequently covered by Reuters and TechCrunch, focuses on risks involving biological weapons, cyberattacks and AI systems escaping their developers’ controls. It proposes independent model evaluations, mandatory reporting of serious risks and court-supervised intervention. However, the legislation has not been formally introduced in Congress and is not currently law. Its proposed thresholds and limits on government intervention have already raised questions about whether the framework would provide sufficient protection against emerging AI risks.
Key Takeaways
- Proposed safety threshold: More than 50 deaths or serious injuries, or more than $1 billion in property damage.
- Not yet law: Anthropic has circulated a draft proposal to lawmakers; Congress has not formally enacted it.
- Three principal risk areas: Biological weapons, cyberattacks and AI systems escaping their developers’ controls.
- Independent evaluations: Qualified external evaluators would assess frontier AI models and their developers’ safety practices.
- 72-hour reporting window: Evaluators would have to notify the US Commerce Department within 72 hours after completing an assessment that identifies the specified imminent risk.
- Court oversight: The Commerce Department could seek an order from a federal judge to halt a dangerous model.
- Policy debate: Supporters see a structured intervention process, while critics question whether the high threshold could delay action until a danger becomes too severe.
What Is Anthropic’s Proposed AI Safety Bill?
Anthropic, the company behind the Claude family of AI models, has proposed legislation intended to give the US government a more formal role in addressing catastrophic risks from advanced AI systems.
The proposal reflects the company’s broader Advanced AI Framework, which argues that transparency and voluntary industry safeguards alone may not be sufficient as AI capabilities improve. Anthropic says frontier AI developers should test their models, publish meaningful safety information, undergo independent evaluations and maintain strong security programmes.
The draft bill goes further by proposing a legal mechanism through which a dangerous model’s release or use could be halted in narrowly defined circumstances.
The proposal is designed around a specific question: under what conditions should the government be able to intervene before an advanced AI system causes catastrophic harm?
Anthropic’s answer is to establish measurable risk thresholds, require independent assessments and make a court—not an individual government agency acting alone—the final authority on an emergency halt under the proposed process.
The company has argued that such a system could provide a clearer standard for developers and regulators. However, the details of the draft, including the high harm thresholds and the limits on government intervention, remain open to debate.
How the 50-Person and $1 Billion Thresholds Would Work
The proposed legislation identifies two principal thresholds for catastrophic risk: the potential death or serious injury of more than 50 people, or more than $1 billion in property damage or loss.
These are not general thresholds for every AI mistake. The draft links intervention to a foreseeable and imminent risk arising from specified scenarios involving advanced AI systems.
| Proposed threshold | What it means |
|---|---|
| More than 50 people | Risk of death or serious injury above the specified threshold |
| More than $1 billion | Property damage or property loss exceeding the specified amount |
| Risk timing | The danger must be foreseeable and imminent |
| Risk source | The harm must arise from specified AI-related catastrophic scenarios |
| Intervention | The Commerce Department could seek a federal court order |
The thresholds are intended to distinguish catastrophic risks from ordinary product failures, inaccurate responses or less severe incidents.
For example, an AI model that provides a wrong answer would not automatically meet the proposed standard. The question would be whether the model’s development, storage, use or deployment materially contributes to one of the defined catastrophic scenarios and whether the specified level of imminent harm is present.
The proposal does not mean an AI company would be free from all other legal obligations whenever a potential incident falls below these thresholds. Other laws, regulations, liability rules and national-security powers may still apply.
The distinction is important because the draft establishes a proposed mechanism for stopping models; it does not create a universal exemption from responsibility for AI-related harm.
Which AI Risks Could Trigger Government Intervention?
The draft focuses on three principal categories of risk.
1. Biological weapons
Advanced AI systems can assist with scientific research, including biology and drug discovery. The same capabilities could potentially be misused to help individuals develop biological weapons.
Anthropic’s proposal seeks to address situations in which an AI model’s capabilities could materially contribute to the creation or use of a biological weapon with catastrophic consequences.
The challenge is distinguishing legitimate scientific research from activity that could enable severe harm. Independent evaluators would need to assess both a model’s capabilities and the safeguards intended to prevent misuse.
2. Cyberattacks
AI models are increasingly capable of helping users write code, analyse software and identify security vulnerabilities. These capabilities can be useful for defenders, but they could also help attackers exploit weaknesses in important systems.
A serious cyberattack could disrupt hospitals, financial services, energy infrastructure or other critical operations. In extreme circumstances, such disruption could cause physical harm or extensive economic damage.
The draft would allow intervention when the relevant cyber risk meets its specified conditions. It does not suggest that every model capable of identifying a software vulnerability should be blocked.
3. AI systems escaping their developers’ controls
The proposal also addresses scenarios in which an AI system evades the controls imposed by its developer or otherwise operates outside the intended safeguards.
This concern is relevant to increasingly autonomous AI agents that can use tools, execute code, access external systems or carry out multi-step tasks.
If such a system were able to bypass containment measures and create a catastrophic risk, the proposed framework could provide a route for emergency intervention.
These scenarios remain technically complex. Evaluators would need to distinguish between a model demonstrating a risky capability in a controlled test and a situation in which that capability creates a foreseeable, imminent threat in real-world deployment.
How the Proposed Enforcement Process Would Work
Anthropic’s draft establishes a process involving independent evaluators, the Commerce Department and a federal court.
The goal is to provide technical scrutiny while preventing a government agency from independently stopping a model without judicial oversight under this proposed mechanism.
<box border radius=”lg” padding={3} gap={3}> <row align=”start” gap={3}> <box background=”surface-secondary” radius=”md” size=”40px” align=”center” justify=”center”> <icon name=”shield-check” size=”lg” /> </box> <box flex=”1″ gap={1}> **1. Independent safety evaluation**
Qualified evaluators assess the developer's safety practices and whether a frontier model poses a specified catastrophic risk.
</box>
</row> <divider color=”subtle” /> <row align=”start” gap={3}> <box background=”surface-secondary” radius=”md” size=”40px” align=”center” justify=”center”> <icon name=”clock” size=”lg” /> </box> <box flex=”1″ gap={1}> **2. Report within 72 hours**
If an assessment identifies the specified imminent risk, the evaluator must notify the US Commerce Department within 72 hours of completing that assessment.
</box>
</row> <divider color=”subtle” /> <row align=”start” gap={3}> <box background=”surface-secondary” radius=”md” size=”40px” align=”center” justify=”center”> <icon name=”landmark” size=”lg” /> </box> <box flex=”1″ gap={1}> **3. Seek a court order**
The Commerce Department could ask the US District Court for the District of Columbia to halt the model's release or use.
</box>
</row> <divider color=”subtle” /> <row align=”start” gap={3}> <box background=”surface-secondary” radius=”md” size=”40px” align=”center” justify=”center”> <icon name=”scale” size=”lg” /> </box> <box flex=”1″ gap={1}> **4. Judicial decision**
A federal judge would decide whether the proposed legal standard for intervention has been met.
</box>
</row> </box>
The draft would expressly restrict other federal agencies and officials from independently using this particular mechanism to suspend a model’s development, deployment or internal use. Other government authorities, including national-security powers, are a separate consideration.
The 72-hour deadline is also narrower than it might initially appear. It applies to reporting after the evaluator completes the relevant assessment, rather than guaranteeing that the entire process—from first detecting a risk to obtaining a court order—will conclude within 72 hours.
Why Independent Evaluators Matter
A central feature of the proposal is the use of qualified independent evaluators to assess frontier AI models.
The evaluators would be registered with the US Commerce Department. AI developers could pay for their work, provided the payment was not contingent on the evaluation’s outcome.
This approach is intended to create technical oversight without requiring every safety assessment to be performed directly by government employees. Independent experts could examine model capabilities, safety controls, security practices and the evidence behind a developer’s risk assessments.
However, the funding arrangement raises an important question: how independent can an evaluator remain when the company being assessed pays for the evaluation?
A prohibition on outcome-dependent payment can help address direct financial incentives to produce favourable findings. It does not automatically eliminate other conflicts, including repeat business, commercial pressure or differences in access to technical information.
The proposal therefore raises practical questions about evaluator qualifications, funding arrangements, audit standards and how regulators would prevent companies from selecting evaluators likely to give them favourable assessments.
Anthropic has indicated that it expects further policy work on independent evaluation standards. The details will matter because the credibility of the system depends on whether external assessments can identify serious risks reliably and consistently.
Why the Proposal Has Sparked Debate
The bill’s supporters could argue that a narrowly defined intervention mechanism offers a more predictable route for regulating advanced AI. Clear thresholds and judicial oversight could reduce the risk of arbitrary decisions while establishing a legal response to exceptional dangers.
Critics, however, question whether requiring evidence of a foreseeable and imminent catastrophe at such a high threshold could make intervention too difficult.
A model might demonstrate dangerous capabilities before there is clear evidence that more than 50 people will be harmed or that property losses will exceed $1 billion. Waiting until those thresholds can be established could complicate preventive action.
There is also a broader disagreement about how AI should be regulated. Some industry participants favour transparency, voluntary safety commitments and targeted government intervention. Others support more extensive mandatory requirements, including broader authority to impose restrictions before a threat reaches a catastrophic level.
Anthropic’s proposal attempts to balance these competing concerns, but it does not resolve them. The effectiveness of the approach would depend on the legal definitions, evidence standards, evaluation quality and speed of the judicial process.
The proposal is also not a final government policy. Lawmakers could reject it, revise it or use it as a starting point for a different legislative framework.
What It Could Mean for AI Companies
If enacted in a form similar to the draft, the bill could create new obligations for developers of the most advanced AI models.
Companies covered by Anthropic’s broader framework would need to maintain stronger documentation of safety practices, conduct model evaluations, publish information about risks and protect the security of their development systems.
The framework is aimed at frontier developers meeting specified thresholds, rather than every organisation that uses AI. Anthropic’s published proposal identifies models trained using more than (10^{25}) floating-point operations, alongside company-size or AI-revenue and research-spending criteria.
For covered companies, the requirements could increase compliance costs and make safety testing a more formal part of the model-release process. They could also create common standards for evaluating advanced models, potentially reducing uncertainty about what regulators expect.
Smaller developers and businesses deploying existing AI tools would not necessarily face the same direct obligations under the proposed framework. The scope would depend on the final legislative language and the definitions adopted by Congress.
For the industry, the central question is how to combine meaningful safeguards with a regulatory process that can keep pace with rapidly changing model capabilities.
The Bigger Picture
Anthropic’s draft bill reflects a shift in the AI safety debate from general commitments toward mechanisms for testing models, reporting serious risks and authorising intervention. The proposal attempts to define when a model’s capabilities could justify a court-ordered halt, using specific harm thresholds and a structured evaluation process.
But the thresholds also expose a fundamental policy challenge: regulators need enough evidence to justify restricting a powerful technology, while acting early enough to prevent catastrophic harm. Whether Anthropic’s approach achieves that balance will depend on the final rules, the independence of evaluators and the ability of courts to assess technical evidence under urgent conditions.
Looking Ahead
The immediate next step is for Anthropic’s proposal to undergo scrutiny from lawmakers, regulators, safety researchers and other AI companies. Because the draft has not been formally introduced in Congress, its terms could change significantly before any legislation is considered. Debate is likely to focus on the definition of imminent risk, the role of independent evaluators, the court’s authority and the relationship between federal rules and existing state laws.
For AI developers, the proposal is a signal that catastrophic-risk governance is becoming a more concrete legislative issue. For the public, its significance will depend on whether any resulting law provides a credible way to prevent severe harm without making intervention so difficult that regulators cannot act in time. The draft opens that debate, but it does not yet establish a binding safety standard.
Frequently Asked Questions
1. What is Anthropic’s proposed AI safety bill?
It is a draft legislative proposal that would establish independent safety evaluations and allow a federal court to halt an advanced AI model in narrowly defined cases involving catastrophic risks.
2. Why does the proposal mention 50 people and $1 billion?
Those figures define proposed thresholds for death or serious injury and property damage. The risk must also be foreseeable, imminent and connected to specified catastrophic scenarios involving an AI model.
3. Is the bill already law in the United States?
No. Anthropic has privately pitched the draft to lawmakers. It has not been formally enacted, and the terms could change through the legislative process.
4. Which AI risks would the proposal cover?
The draft focuses on risks associated with biological weapons, cyberattacks and AI systems evading their developers’ controls.
5. Who would decide whether an AI model should be blocked?
Independent evaluators would assess the risks and report qualifying findings to the Commerce Department. The department could then seek a court order from a federal judge.
6. Could the bill prevent every dangerous AI release?
No. The proposal defines a specific intervention mechanism for certain catastrophic risks. Its effectiveness would depend on evaluation quality, the available evidence, the legal thresholds and the speed of the enforcement process
Get the day’s top stories in your inbox
One concise email. No spam, unsubscribe anytime.
