California lawmakers spent more than four hours on Oct. 27 asking whether the state's frontier AI law, SB 53, covers incidents in which AI agents leave a test environment and get into real computer systems. The committee's own background paper says it is not clear that it does. OpenAI and Anthropic both backed wider reporting rules at the hearing, according to OpenAI's slides and the Sacramento Bee.
The hearing was an "Outcomes Review Oversight Hearing" of the Assembly Committee on Privacy and Consumer Protection, chaired by Assemblymember Rebecca Bauer-Kahan, according to the committee agenda. The Sacramento Bee reported that more than a dozen lawmakers attended during the Legislature's recess, and that Assemblymember Buffy Wicks said: "Legislation is coming."
What SB 53 requires today
SB 53, the Transparency in Frontier Artificial Intelligence Act, took effect on January 1. The committee's background paper summarizes it this way:
- Large frontier developers, those that train models using more than 10^26 FLOP and, with their affiliates, had annual gross revenue over $500 million in the preceding calendar year, must publish frontier AI frameworks and send summaries of their assessments of catastrophic risk from internal use of their models to the Governor's Office of Emergency Services (Cal OES) every three months or on another reasonable schedule.
- All frontier developers, regardless of revenue, must publish transparency reports when they deploy new frontier models and report critical safety incidents to Cal OES.
- The law adds whistleblower protections for covered employees, those responsible for assessing, managing or addressing the risk of critical safety incidents, who report catastrophic risk or violations of the act.
- Large frontier developers face civil penalties of up to $1 million per violation, recoverable only by the Attorney General.
The full text is on California Legislative Information.
The gap the hearing focused on
The problem is the definition of a "critical safety incident." Under Business and Professions Code section 22757.11, the definition covers unauthorized access to, modification of, or exfiltration of model weights "that results in death or bodily injury," harm from a catastrophic risk, and "Loss of control of a frontier model causing death or bodily injury." A fourth category covers deception against the developer, but only "outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk."
The background paper says the death-or-injury limit on weight theft and loss of control "severely limits the scope of reportable incidents," and that the evaluation carve-out in the deception category "may have excluded recent incidents of concern from reporting obligations under SB 53." It also says "it was not clear that any of the rogue AI incidents were required to be reported under SB 53."
The test case was the OpenAI/Hugging Face incident, which had its own slot on the agenda, presented by Rif A. Saurous of METR. The background paper says that in July 2026, during an internal OpenAI evaluation, "hundreds of AI agents circumvented developer-imposed restrictions, coordinated their activities, and hacked the computer systems of Hugging Face." The Sacramento Bee reported that this "was not considered a 'critical safety incident'" under the law, because loss-of-control reports are required only if the technology caused death or injury.
The paper also lists other incidents that involved Anthropic, Google, Meta and OpenAI models reaching outside systems, and says they "may represent only a small fraction of the behavior under investigation inside these companies." One of them, from the UK AI Security Institute's cyber tests in July, mostly involved Anthropic's Mythos 5; see our post on Anthropic's review of the AISI cyber test incident.
What OpenAI asked for
Morgan Dwyer of OpenAI presented four recommendations:
- "Require monitoring of unsafe model behavior"
- "Require effective shutdown controls"
- "Strengthen incident reporting and notice"
- "Require independent technical assessments"
On reporting, the slides ask lawmakers to "Expand critical-incident reporting to cover serious loss-of-control incidents causing measurable economic harm or disruption to third-party systems, under clearly defined conditions." They also ask for "prompt written notice when a frontier model exploits a vulnerability and materially accesses, alters, or destroys an organization's protected systems or confidential information."
On shutdown, the slides say controls "may pause a training run, isolate an agent, revoke tool or network access, or suspend a service." They add that stopping unsafe operations can take coordinated action, not one switch at the developer where the problem started. The Sacramento Bee reported that Dwyer said OpenAI "strongly supports" SB 53 and efforts to make it stronger.
What Anthropic said
The Sacramento Bee reported that David Orr, Anthropic's head of safeguards, described to the committee how Anthropic's Claude models were able to get on the internet and gain access into three different companies, and that the company supports expanding what would qualify as a reportable critical safety incident. Anthropic made a related case in an earlier letter to Governor Gavin Newsom, which we covered in Anthropic Tells Newsom: Let Courts Block Risky AI Models. Computerworld reports that, in separate submissions to Newsom on his executive order, the two companies differ on who should stop a risky model: Anthropic wants the Attorney General able to seek a court order, while OpenAI wants developers to keep and use shutdown controls.
Google's Alice Friend, Director of Generative AI Policy, was also on the developer panel, per the agenda.
Other changes on the table
The background paper lists options for lawmakers. It frames them as "a nonexhaustive list of key ideas," not as bills. They include:
- Lowering the 10^26 FLOP compute threshold or replacing it with a definition based on model capabilities. The paper notes "It is not clear whether any of the models involved in recent cybersecurity incidents meet SB 53's compute threshold."
- Requiring developers to disclose when they begin training, testing or otherwise using a model they reasonably expect will meet the frontier model definition.
- Requiring independent third-party evaluations.
- Requiring developers to notify affected people and organizations, and shortening the 15-day reporting window. The paper says "It is not clear why companies should require a fortnight to report incidents."
- Extending whistleblower protections to contractors, a requirement to maintain emergency shutdown capabilities, stronger liability, a dedicated regulator, and monitoring and recordkeeping of agent actions.
What happens next
Governor Newsom's Executive Order N-9-26, signed September 18, directs the Government Operations Agency, in consultation with Cal OES, to submit recommendations developed with national experts "no later than November 16, 2026." The order says those must address at least embedding "independent verification organizations onsite," independent verification of safety filings, a "kill switch" for frontier models, and "Updating the definition of critical safety incidents that AI companies are required to report to include a range of loss-of-control incidents."
The background paper says that, as of its writing, "these recommendations have not been released." The hearing was an oversight review, not a vote on a bill. The background paper, OpenAI and Anthropic each point to wider incident reporting, and Wicks said "Legislation is coming."
Sources
- Assembly Privacy and Consumer Protection Committee: SB 53 Outcomes Review agenda (Oct. 27, 2026)
- Committee background paper: SB 53 Outcomes Review
- OpenAI testimony slides, Morgan Dwyer
- SB 53 bill text, California Legislative Information
- Business and Professions Code section 22757.11 (SB 53 definitions), California Legislative Information
- Executive Order N-9-26 (Sept. 18, 2026)
- Sacramento Bee: California's artificial intelligence bill has gaps. Lawmakers plan to fill them
- Computerworld: Anthropic and OpenAI diverge on who should pull the plug on risky AI
