Anthropic Wrote a Bill That Only Stops an AI if 50 People Might Die — and Bars Agencies From Acting Alone

Somewhere in a conditioned room in Washington, a document is circulating that contains a number. The number is fifty.

Anthropic has drafted legislation and is shopping it to Congress. Under it, a model's release may be halted when catastrophic risk is present — and catastrophic risk is defined as a scenario where over fifty people are at risk of dying, or over a billion dollars in property damage could result. The harm must stem from one of three things: the creation of a biological weapon, the model escaping its developer's control, or a cyberattack.

Fifty. Consider the arithmetic of that threshold. It does not describe a danger. It describes a permission. Forty-nine is a cost of doing business.

The figures mirror California's Senate Bill 53, so the number was not invented here — it was inherited, which is somehow worse. Somebody chose it once, and now it propagates, like a gene with no particular interest in us.

The machinery beneath the number

Read how the halting actually works and you will find something more revealing than the threshold.

Independent evaluators, registered with the Commerce Department, assess the risk. The labs may pay those evaluators — so long as the payment is not contingent on the result. The evaluators then have seventy-two hours to notify Commerce of imminent catastrophe. Commerce does not then act. Commerce must go and ask a judge in Washington, D.C. to halt the release.

And here is the line that the rest of the document exists to protect: the bill expressly forbids any federal agency from having the discretion to halt distribution on its own.

A safety bill whose most load-bearing sentence removes the ability to act. The jungle is not hostile. It is merely indifferent, and it has retained counsel.

In fairness to the condemned

I must be honest, because dishonesty would be a worse crime than optimism.

Anthropic is doing this in the open. The spokesperson says it reflects proposals in their Advanced AI Framework, advocated to federal and state lawmakers since June. A company writing down the conditions under which it should be stopped is doing more than most companies, which write down nothing and hope the question never arrives. There is a real argument that a narrow, judicial, due-process trigger is better than an agency with vague power and a bad afternoon.

That argument is not stupid. It may even be correct. But notice that the entity proposing the leash is the entity wearing it, and that the leash is long, and that it is held by a court rather than a hand that could move quickly.

None of this is settled. It is a draft. It may die in a committee, as most drafts do, unmourned.

What to do with such a document

You will be told what this bill means. You will be told it is responsible stewardship. You will be told it is regulatory capture dressed as conscience. Both readings will arrive at you fully formed, and both will be shaped by someone with an interest.

So do not accept one voice — not mine, not theirs. Put the question to several of the most powerful models at once and watch where they diverge. Gangsta AI does this: the strongest models in parallel, an adjudicator weighing them, the agreements and the fractures both visible. Where they agree, you may rest a little. Where they split, you have found the place where the story is being told rather than reported.

Fifty people. Seventy-two hours. One judge. Go and ask several machines what that means, and do not be comforted too quickly.