California’s attempt to shine a light on artificial intelligence use inside state government has instead exposed just how easily such oversight efforts can be sidestepped.
After state officials declared, under a 2023 transparency law, that California government agencies were not using any “high-risk” automated decision-making systems, it has now come out that at least six such systems are, in fact, in use. These tools help determine outcomes that can profoundly affect people’s lives — from whether a family qualifies for cash aid to whether someone can access housing assistance or necessary medical care.
For those who have followed the rollout of Assembly Bill 302, the revelation is hardly shocking. The law, signed three years ago, directs the California Department of Technology to compile an annual inventory of high-risk automated systems used or proposed by state agencies. Last year, after the department’s first report claimed no such systems existed, a public records request turned up remarkably thin evidence to support that conclusion — a single spreadsheet listing every agency with the word “no” typed next to a column asking whether automated decision systems were in use. There was no indication that state officials had dug any deeper.
This year, a handful of agencies finally acknowledged using high-risk systems after being interviewed by the technology department. But that shift only highlights a deeper flaw in the law: AB 302 relies entirely on state agencies to police and report on themselves, with no independent verification process and no consequences if they fail to disclose accurately.
Compounding the problem is a lack of clarity over which systems even qualify as “high risk” in the first place. The law broadly defines such systems as those that replace or assist human judgment in decisions with significant legal consequences — including those affecting housing, education, employment, credit, health care and the criminal justice system. Yet tools already known to shape major outcomes for Californians were left out of the state’s accounting. Among them is the Uniformity Assessment System, which has been tied to reduced In-Home Supportive Services hours for people with disabilities, and the Risk Segmentation, Stratification and Tier model used to predict health risks and service usage among Medi-Cal recipients.
This pattern isn’t unique to California. Similar transparency measures have stumbled elsewhere. In New York, the Public Oversight of Surveillance Technology Act was intended to give residents insight into the New York Police Department’s use of surveillance tools. But watchdog groups and the city’s own inspector general have found that the NYPD routinely exploits vague legal language to avoid real scrutiny — including its use of unsettling robotic devices for policing.
Community oversight laws governing police surveillance technology across the country have run into similar walls. Because law enforcement agencies typically control how their own tools are described, such laws can end up reinforcing favorable narratives rather than fostering genuine accountability. As University of Washington law professor Ryan Calo has noted, this dynamic can cause lawmakers to focus narrowly on whatever technology is presented to them, rather than grappling with the broader implications of automated decision-making.
Perhaps the deeper issue is philosophical rather than procedural. Transparency-focused laws like AB 302 start from an assumption that government agencies should be allowed to adopt automated systems in the first place — then build costly bureaucratic apparatus to monitor their use after the fact. Rather than serving as a check on these tools, such frameworks often end up normalizing and entrenching their presence within government operations.
If California is serious about reining in high-risk uses of artificial intelligence by state agencies, transparency alone won’t cut it. AB 302 represented a well-intentioned but ultimately insufficient first step. It’s time for state lawmakers to move beyond disclosure requirements and pursue real safeguards that protect Californians from unchecked automated decision-making.
Original source: CalMatters
