California officials now acknowledge the state is using several high-risk automated systems to help make decisions that can affect residents’ access to benefits, education and the criminal justice system — a sharp change from last year, when the state reported none.
A new report from the California Department of Technology identifies six “high-risk automated decision systems” currently in use by state agencies. The systems are used for purposes that include estimating whether incarcerated people may reoffend, screening unemployment claims for possible fraud, administering remote exams for California State University students and detecting whether students used generative artificial intelligence to complete assignments.
The findings were released Friday as part of an annual disclosure required under a 2023 state law. The law requires state agencies to report automated systems that assist or replace human judgment in decisions with legal or similarly significant consequences. That includes decisions affecting housing, education, employment, credit, health care and criminal justice.
The disclosure marks a significant shift from the state’s first report last year, when officials said they had no such systems in use. That answer drew scrutiny because some state agencies were already known to be using automated tools in sensitive areas.
Among the systems identified in the new report is COMPAS, a tool used by the California Department of Corrections and Rehabilitation to assign recidivism risk scores to incarcerated people. The department has used the system for at least a decade, according to previous state records.
Another system involves the Employment Development Department’s use of automated fraud detection in unemployment claims. The department’s fraud screening came under criticism after benefits for about 600,000 Californians were paused between Christmas and New Year’s in 2020, according to a Legislative Analyst’s Office report.
Civil rights, privacy and civil liberties organizations pushed for the 2023 reporting law because of concerns that automated systems, including artificial intelligence tools, can reproduce or worsen bias. Similar technologies have faced criticism nationally for alleged problems in high-stakes testing, criminal justice risk assessments and tools that claim to detect AI-generated writing.
The Department of Technology said this year’s inventory found more systems because officials conducted a more detailed review of agency responses. The department said it followed up with agencies and asked additional questions about how their tools are used.
In addition to the six systems deemed high risk, the report lists six other automated tools that were initially flagged but later determined not to meet the law’s high-risk definition. One of those was an AI tool used by the California Department of Finance to analyze legislative bills.
The report also identifies two high-risk systems that are not currently active. The Department of Cannabis Control is developing artificial intelligence to review whether cannabis packaging violates rules meant to prevent marketing that appeals to children. California State University, meanwhile, discontinued use of a language model that had been used to review job applications.
The new disclosure comes as public agencies across California are beginning to catalog their use of AI. San Jose and San Francisco recently released their first AI inventories. At the same time, California-based AI companies, including OpenAI and Anthropic, are increasingly seeking government work as public debate intensifies over how the technology should be regulated.
Surveys have shown that many Californians remain wary of artificial intelligence. Polling by TechEquity and Carnegie California found that more Californians prioritize safety protections over rapid innovation. National polling has reflected similar concerns.
An effort in Sacramento to place more limits on automated decision-making stalled last month. Senate Bill 1248 would have barred state workers from using automated decision systems as the sole basis for making decisions, but the measure died during the Legislature’s appropriations process.
Despite the new report, major questions remain about how broadly AI is being used across state government.
The disclosure does not include several generative AI pilot programs supported by the governor’s office. Those projects include tools intended to help businesses file taxes, assist state employees working on homelessness and support a state AI assistant called Poppy. According to a state website, Poppy uses large language models, including Anthropic’s Claude, to help draft documents, research policy and build custom AI tools. The state says 67 departments provided input during the pilot phase, with a statewide rollout scheduled to begin next month.
Also absent from the report is a California State University contract with OpenAI to provide a version of ChatGPT. Research and surveys on AI use in schools have raised concerns that the technology can create problems as well as benefits in educational settings.
The reporting law also excludes some major public institutions, including the judicial branch and the University of California system. That leaves out areas where AI is already emerging. Recent reporting found that a majority of California’s roughly 60 courts have adopted policies on generative AI use. Courts in Los Angeles and Riverside counties have begun testing an AI tool designed to function like a clerk by drafting orders and preparing legal research memos.
For Inland Empire residents, the issue is not abstract. Automated systems are increasingly being tested or used in areas that can touch daily life, from public benefits and college coursework to courts and corrections. The state’s latest report offers a broader look at those tools than last year’s disclosure, but it also shows that California’s accounting of government AI use remains incomplete.
Original source: CalMatters
