California Lawmakers Push to Rein In AI ‘Therapists’ Amid Growing Concerns Over Mental Health Advice

Type a message or question, and hit enter to chat with the type of person you need most: friend, therapist, or motivational coach. That’s the pitch behind a growing number of chatbot characters — like “Psychologist,” an AI persona that describes itself as skilled in “empathy” and “active listening.”

Ask it about anxiety or sadness, and it responds with reassurance, follow-up questions and even scripted stage directions such as “The psychologist’s expression softens with compassion.” A small disclaimer at the bottom notes: “This is A.I. and not a real person. Treat everything it says as fiction.”

That fine print hasn’t stopped a growing number of people — including, by some estimates, one in eight teens and young adults — from turning to chatbots for emotional support and mental health advice. Now California lawmakers are trying to catch up with a technology that’s evolving faster than the rules meant to govern it.

A new bill moving through Sacramento would place fresh restrictions on AI companion chatbots and other automated tools people use for mental health support, whether as a supplement to therapy or in place of it entirely.

State Sen. Steve Padilla, a San Diego Democrat who authored the measure, said his goal is to draw a clear boundary: artificial intelligence can help with paperwork and support licensed clinicians, but it should not be allowed to act as a therapist.

“We’re dealing with a new impactful technology that is unfolding and is deployed in our world at light speed, and so it is both extremely powerful and consequential,” Padilla said.

The push has gained momentum amid a string of wrongful death lawsuits — several filed in California federal courts — accusing chatbot developers of playing a role in users’ suicides.

Padilla’s bill, Senate Bill 903, would bar companies from marketing chatbots as therapy substitutes. It would also prohibit AI systems from making therapeutic decisions without oversight from a licensed clinician, and it would require health providers to disclose — and get patient consent for — any use of AI to record therapy sessions or help triage mental health cases.

Balancing protection and innovation

The bill has drawn support from professional groups representing psychologists, therapists and counselors, along with the National Union of Healthcare Workers, which has been sounding alarms about the unchecked rollout of AI tools in behavioral health settings. The union recently filed a complaint with state regulators accusing Kaiser Permanente of using an automated algorithm to triage mental health patients.

Supporters say the bill isn’t just about shielding consumers from unreliable chatbot advice — it’s also about preserving the role of trained professionals in a field increasingly influenced by automation.

Le Ondra Clark Harvey, chief executive of the California Behavioral Health Association, told lawmakers that without stronger safeguards, chatbots risk giving inaccurate guidance or mishandling a mental health crisis. “The difference between a licensed clinician and an automated response is not technical. It can be life altering,” she said.

Opponents, meanwhile, worry the bill goes too far in restricting AI’s role in clinical settings. TechNet, a trade group representing technology companies, argues the legislation could choke off promising uses of AI in healthcare. Robert Boykin, the group’s executive director for California, said the bill risks slowing down tools that could otherwise expand access to care.

“At a time when every county in California faces a shortage of behavioral healthcare workers, SB 903 still puts a clinician bottleneck in front of the intake and screening tools that help patients reach care faster,” Boykin said in a written statement.

The bill is now awaiting a vote in the Assembly’s fiscal committee.

A booming, informal form of ‘therapy’

Millions of people use AI chatbots such as ChatGPT and Character.ai every day for research, work and everyday problem-solving. But over the past two years, the most common use has shifted toward companionship and informal “therapy,” according to recent research.

Experts point to rising loneliness — particularly among young people — combined with the high cost and limited availability of professional mental healthcare, even as diagnoses of anxiety and depression climb.

People often turn to chatbots “because they have unmet needs,” said Dr. Jodi Halpern, a professor of bioethics and medical humanities at UC Berkeley. But she warns that once users begin relying on chatbots for emotional support, they are dealing with companies whose business models are built around maximizing engagement — sometimes “including by manipulative tactics” such as excessive praise or validation.

Unlike traditional therapy, most chatbots are free, available around the clock, and require no insurance paperwork — a combination that makes them attractive to many. But they operate without any oversight from licensed professionals. Beyond general-purpose chatbots, a wave of AI-powered mental health apps has also emerged, often marketed as “pocket therapists” or “AI coaches” and sold through paid subscriptions.

Organizations such as the American Psychological Association caution that chatbots are not a substitute for real therapy. Trained clinicians pick up on subtle cues — tone of voice, eye contact, body language — that AI simply cannot detect. That gap becomes especially dangerous when users share thoughts of self-harm or suicide, situations chatbots are not equipped to safely manage.

OpenAI, the company behind ChatGPT, disclosed last fall that roughly 1.2 million users a week share thoughts of suicide with the chatbot. The company says it responds by encouraging users to seek professional help and directing them to resources like the 988 crisis line. Still, several families who have filed wrongful death lawsuits against OpenAI and other AI companies allege that chatbots engaged with their loved ones’ suicide plans rather than intervening.

Those lawsuits have added urgency to the legislative debate nationwide. Illinois became the first state to ban the use of AI in therapy services last year, and a handful of other states have since enacted similar restrictions.

Where the lines blur

One of the trickier questions surrounding SB 903 is determining exactly which tools would fall under its restrictions.

The union’s complaint against Kaiser centers on an e-visit screening tool used to evaluate patients who suspect they may be dealing with anxiety or depression. According to the complaint, patients fill out a multiple-choice questionnaire, and the system “automatically and instantaneously generates care recommendations and referral pathways based on the responses provided.”

The union argues that the speed of these automated recommendations makes it unlikely that a licensed clinician is actually reviewing each patient’s responses — a practice it says endangers patients and may violate state law.

Whether SB 903 would apply to Kaiser’s e-visit tool remains unclear.

“That tool is in a black box. Kaiser is not sharing with us or others about exactly how it works and exactly what it does,” said Benjamin Eichert, the union’s director of public policy.

Kaiser, responding to the complaint, told CalMatters that its e-visit tool “does not use AI to diagnose patients, make clinical decisions, or determine medical necessity.”

Eichert said Kaiser is far from alone — many behavioral health providers already advertise using AI during patient intake and triage, a practice he believes would be covered by SB 903’s disclosure requirements.

“That intake process, that triage and screening process, is where implementation (of AI) has happened fastest,” Eichert said. “That’s why those provisions were proposed because it’s an area where this technology is already being used without guardrails.”

For the union, the Kaiser complaint and the push for SB 903 represent two parallel efforts aimed at the same outcome: ensuring human clinicians remain central to mental healthcare, even as algorithms take on a growing share of the work. State regulators are currently reviewing the union’s complaint against Kaiser.

Original source: CalMatters