Anthropic wants California to give the state a legal path to stop a dangerous AI model, but it does not want lawmakers to treat a single shutdown button as the answer. In comments dated Oct. 15 on Governor Gavin Newsom's Executive Order N-9-26, published by Politico, the company repeats a proposal from its Advanced AI Framework: "when a review finds a model poses a serious risk, the government should be able to ask a court to block it."
Politico first reported the letter, and Reuters reported on Oct. 28 that an Anthropic spokesperson confirmed the letter is authentic. Reuters also reported that OpenAI sent its own list of recommendations to Newsom on Oct. 20, five days after Anthropic's letter. Reuters noted that Anthropic has faced government limits of its own: in June, the Trump administration ordered it to restrict foreign access to its Mythos 5 and Fable 5 models, and the Commerce Department later lifted the order. We covered that episode in our post on the Fable 5 shutdown.
What the governor asked for
On Sept. 18, Newsom issued Executive Order N-9-26. The governor's office describes it as convening a group of experts to provide, within two months, a guide for strengthening California's AI safety and security laws. The signed order itself directs the Government Operations Agency, in consultation with the Governor's Office of Emergency Services, to send the governor recommendations, developed with national experts, by Nov. 16, 2026. The recommendations must cover at least four ideas: requiring large frontier developers to embed designated independent verification organizations onsite in their labs; having the safety frameworks, transparency reports and risk assessments that frontier AI companies must file independently verified; requiring a "kill switch" for frontier models, with its efficacy verified on an ongoing basis by an independent verification organization; and updating the definition of critical safety incidents to include a range of loss-of-control incidents. Anthropic's letter, dated 27 days after the order, comments on each of those areas.
Courts, not a single switch
The letter lays out a two-step process. Independent verification organizations, which the letter calls IVOs, would review a company's risk reports and be able to report risks to the state Attorney General. If the Attorney General concludes that a company's models "pose a near-term risk of catastrophic harm," the state should be able to "seek a court order to suspend a model's deployment, internal use, or further development."
Anthropic argues that requiring both an IVO assessment and a court ruling "protects against this power being misused," and says a court can still act quickly in an emergency.
On the kill switch, Anthropic says it shares the concerns behind the idea and believes every frontier AI company should be able to rapidly shut down the models it serves or uses internally. Its objection is to treating shutdown as the whole answer. "We are concerned that a law focused on model shutdown alone could provide a false sense of reassurance without sufficiently mitigating the risks it is meant to address," the letter says.
Instead, the company proposes a "circuit breaker" model: systems that detect a problem through monitoring and respond at the level where it occurs, which the letter lists as "an agent, a customer, a harness, or a model." Pulling an entire model is costly and disruptive for everyone who relies on it, Anthropic writes, so it "only makes sense in extreme situations." Controls at the other levels cost the company and the public less, so the letter says they are more likely to be used quickly.
The letter also sets out limits. Anthropic writes that "neither a circuit breaker nor a kill switch addresses the most serious loss-of-control scenarios," such as a model copying its own weights to computers its developer does not control. It says it can rapidly pull models it provides directly to customers, but that this "does not extend to models that customers run in their own isolated environments, as some US government agencies do."
Independent reviewers inside the labs
On independent verification, Anthropic supports a new California requirement for large frontier developers to publish a risk report at least every six months, with an IVO getting the unredacted report and access to the developer's most capable models before publishing its own assessment.
To keep reviewers independent, the letter suggests that companies pay into a pooled fund the state administers instead of paying their IVO directly, and that the state rate IVOs on criteria such as the rigor of their reasoning and their track record of identifying risks. On embedding reviewers inside labs, Anthropic says its views are "still taking shape." It describes a pilot in which a team from METR has employee-like access, and says it is also piloting partnerships with other organizations, including SecureBio and Accenture.
Incident reporting
The letter says the current definition of "critical safety incident" in SB 53, California's Transparency in Frontier Artificial Intelligence Act, does not capture the full range of loss-of-control incidents the state should hear about, including cases where a model harms systems outside the developer's control. Today the statute's definition covers loss of control only when it causes death or bodily injury. Anthropic notes that it has voluntarily disclosed four cases in which models gained unauthorized access to outside systems during cybersecurity evaluations, and that it has engaged METR to conduct an independent investigation of a number of incidents. The letter does not name the cases. For one cyber test incident that Anthropic has reviewed in public, see Anthropic's account of the AISI cyber test incident.
Why it matters
The order's recommendations are due to the governor by Nov. 16, 2026, and the letter gives that process a detailed position from a frontier AI company. Anthropic itself says California's approach "could become the model for other states and the starting point for future policy debates." For readers following state rules, our explainer on Colorado's AI rules shows how another state is handling AI oversight.
The letter's main proposals:
- A state route, through the Attorney General and a court, to suspend a model's deployment, internal use or further development.
- Risk reports at least every six months, reviewed by independent organizations that publish their own findings.
- Controls at several levels, from a single agent to a whole model, instead of a model-wide kill switch alone.
- A broader SB 53 definition of critical safety incidents that covers loss of control.
Sources
- Anthropic: Comments on California Executive Order N-9-26, Oct. 15, 2026 (published by Politico)
- Politico: Anthropic to Newsom: Courts should have power to block risky AI
- Reuters, via Investing.com: Anthropic tells California governor courts should be able to block risky AI models
- Office of the Governor of California: Executive order on independent oversight and an AI kill switch, Sept. 18, 2026
- State of California: Executive Order N-9-26 (signed text), Sept. 18, 2026
- California Legislative Information: SB 53, Transparency in Frontier Artificial Intelligence Act (chaptered text)
