California admits using high-risk AI — including systems it failed to report last year

This story was originally published by CalMatters. You can sign up for their newsletter here.

A year ago, California officials had to report under a new state law how they used automated systems to make important decisions about people’s lives. 

They said they never did — a startling answer for a number of reasons, sources told CalMatters at the time, including that there were several prominent examples to the contrary.

Now, the state has issued a more expansive answer: It is currently using six automated systems to make consequential decisions about the lives of Californians.

The systems are used to do things like:

• Predict whether incarcerated people will re-offend 
• Evaluate whether unemployment claims are fraudulent
• Remotely administer exams for California State University students 
• Detect when college students use generative AI to write assignments.

That’s according to a report released Friday by the state’s technology department. The report is required under a 2023 law mandating that that state agencies annually disclose their use of “high-risk automated decision systems,” which the law defines as systems “used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.”

The law was pushed by civil rights, privacy, and civil liberties groups concerned about harms from AI-like systems. Numerous such systems have been shown to produce results biased against marginalized groups, including those used for high-stakes testing, predicting recidivism, and detecting AI-generated texts.

CalMatters flagged last year’s report as surprising, noting that the state corrections department had reported using software to predict post-release behavior and that the employment department used a fraud detection system that paused benefits for 600,000 Californians between Christmas and New Years in 2020, according to a Legislative Analyst’s Office report.

Though the report names six high-risk systems in use today, state agencies have used some for several years now. Those include COMPAS, which has been used by the California Department of Corrections and Rehabilitation to assign recidivism scores to inmates for at least a decade.

The technology department said in the report that it found more systems for its report this year because it evaluated responses from state agencies more thoroughly, including by meeting with agencies and questioning them about their systems.

In addition to the six high-risk systems, the department’s report disclosed an additional six systems initially flagged as high risk but later determined not to be. One was AI used for legislative bill analysis by the California Department of Finance. 

The report also notes two high-risk systems that are not currently in use: the Department of Cannabis Control is developing artificial intelligence to analyze whether marijuana packaging violates a law against appealing to children and California State University discontinued use of a language model for reviewing job applications.

Results of the second annual survey come after cities like San Jose and San Francisco released their first AI inventories in recent months. They also come at a time when California-based AI companies like Anthropic and OpenAI are going public and seeking government contracts. Americans are split on whether they trust AI and surveys last year by TechEquity and Carnegie California found that the majority of Californians want safety over innovation. A Gallup poll to evaluate the opinions of Americans found similar results.

Senate Bill 1248, a bill that would have prohibited state employees from using automated decision systems as the sole basis for decisionmaking, was killed last month in the state’s rapid-fire appropriations process.

What’s missing

While the newly-released report shares more information than last year’s, several questions remain about the state’s use of artificial intelligence and other automated systems.

The report does not include generative AI pilot projects underway with support from the governor’s office to do things like help businesses file taxes, support state employees who work on homelessness, and an AI assistant named Poppy that uses language models like Anthropic’s Claude to do things like draft documents, research policy, or build custom AI tools, according to a state website. The website says that 67 state departments provided input during the pilot phase and statewide rollout of Poppy begins next month.

A California State University contract with OpenAI to provide a version of ChatGPT is also not mentioned, though surveys of AI use in educational settings have found that the technology can do more harm than good.

The 2023 law mandating the annual high-risk systems report excludes reporting by a number of state agencies, including the judicial branch and the University of California college system. Reporting by CalMatters last month found that a majority of the roughly 60 courts that operate statewide have adopted generative AI use policies. Courts in Los Angeles and Riverside counties have begun testing an AI tool to act as a clerk, drafting orders and producing research memos.

CalMatters is compiling an inventory of automated decisionmaking systems in use by state and local agencies throughout California in order to provide transparency into how governments are using decisionmaking systems and AI. Know about an AI system in use by a state or local agency? Email khari@calmatters.org.

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.