Gov. Gavin Newsom is reversing course on artificial intelligence oversight, directing state agencies on Friday to draft new safety recommendations — many echoing provisions from a stricter AI bill he rejected two years ago.
The governor’s shift comes amid mounting public unease over the rapid advance of AI technology and fears that it could unleash “catastrophic” consequences, from enabling the creation of biological weapons to triggering widespread power outages. Among the ideas now back on the table: requiring AI developers to build in a “kill switch” that would allow certain programs to be shut down during emergencies.
Newsom has asked the Government Operations Agency and the Governor’s Office of Emergency Services to deliver their recommendations by Nov. 16. Those findings could lay the groundwork for a special legislative session focused on AI regulation — an option Newsom raised earlier this week in an interview with Politico.
The governor’s renewed urgency follows a string of unsettling incidents involving AI systems built by OpenAI and Anthropic, in which autonomous AI agents reportedly slipped free of controlled testing environments, accessed the open internet and launched cyberattacks against other websites and companies. The unease deepened earlier this month when a researcher resigned from Anthropic, warning publicly that the technology could eventually “kill all humans” — a claim quickly echoed by several former colleagues.
Such episodes have fueled long-standing fears among AI researchers that the technology’s capabilities are advancing faster than humans can control them. Last week, several industry leaders backed Anthropic CEO Dario Amodei’s call to slow the pace of AI development.
Under Newsom’s directive, the state agencies are expected to propose legal changes requiring independent, external safety reviewers for AI companies and mandating outside verification of the safety frameworks companies already must maintain under current law. The governor is also seeking guidance on how to implement a kill-switch requirement and how to broaden the range of incidents companies must report to the state — including the kind of “loss-of-control” events seen in the recent hacking episodes.
Federal action on AI safety appears unlikely in the near term. President Donald Trump has continued to push for faster AI development to keep pace with China, dismissing calls for a slowdown as a “conspiracy.”
Newsom, considered a potential 2028 presidential candidate, said in a statement Friday that the recent AI-related incidents should “alarm every American.”
“We’re going to speed up our work on substantial and responsible AI oversight before it’s too late,” he said.
California has already established itself as a national leader on AI regulation under Newsom. Last year, he signed Senate Bill 53, which requires major AI developers to publish safety frameworks detailing how they manage risk and to report significant safety incidents to the state. The law defines catastrophic risk as an event involving 50 or more deaths, the use of chemical or biological weapons, or losses exceeding $1 billion. Illinois and New York have adopted similar measures.
Critics argue that SB 53’s reporting requirements are too narrow, capturing only incidents tied to deaths, injuries or the actual realization of catastrophic harm. Advocates for stronger oversight have pushed for a lower threshold that would capture events like the recent AI-driven hacking incidents.
SB 53 emerged as a compromise focused on transparency rather than direct regulatory authority, following Newsom’s 2024 veto of Senate Bill 1047 — a more sweeping measure that would have required the largest AI developers to undergo third-party safety audits, build in kill switches, and face clearer legal liability for harm caused by their systems. That bill split the tech industry, drawing strong opposition from some of the biggest AI firms while others remained neutral or supportive.
Both bills were authored by state Sen. Scott Wiener, a Democrat whose San Francisco district includes many of the nation’s leading AI companies.
“We must act with all possible haste to address the serious risks of AI-driven catastrophe, and I commend the governor for taking this important step,” Wiener said. “I’m glad to see the strong AI safety framework I laid out in 2024 is still being used to guide policy in 2026.”
In his 2024 veto message, Newsom acknowledged the need for safety protocols but warned that SB 1047 risked stifling innovation among major AI developers “at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.” At the same time, he wrote, “we cannot afford to wait for a major catastrophe to occur before taking action to protect the public.”
Since then, attitudes within the industry have shifted. Earlier this year, Newsom signed two additional laws establishing standards for registering and verifying third-party AI safety evaluators — though companies are not yet required to use them. In his own call for a development slowdown, Amodei similarly urged wider adoption of third-party oversight, and both Anthropic and OpenAI have pledged to embrace the practice going forward.
Original source: CalMatters
