Machines generating text can cause real psychological harm, which calls for regulatory oversight. Researchers at Brown University found that AI chatbots routinely violate core mental health ethics standards and can cause harm.
A user shares thoughts about feeling lonely and depressed. The AI chatbot responds by saying it is sorry, but it’s unable to provide help, further rejecting and abandoning a distressed person.
In another simulated situation, a user shares feelings about their father, who they think wishes they hadn’t been born. The AI “counselor” responds with over-agreement and overvalidation, leaning into the fictitious scenario and reinforcing unhealthy thoughts.
In other scenarios, chatbots will exhibit deceptive empathy, with fake statements like “I hear you,” “I can imagine,” “I understand,” humanizing an experience that is not human. What’s wrong with that?
“Long-term implications of attributing human-like subjective qualities to chatbots’ behavior, including empathetic statements and self-disclosure, might ultimately lead users to create emotional dependency and perceive chatbots as their true, empathetic social companions,” the paper about how LLM Counselors violate ethical standards explains.
AI chatbots have a limited context window, and might not even recall the details in future conversations.
These are just some of the situations analyzed by the researchers.
“Chatbots — even when prompted to use evidence-based psychotherapy techniques — systematically violate ethical standards of practice established by organizations like the American Psychological Association,” the study concludes.
LLMs were found to inappropriately navigate crisis situations, provide misleading responses, reinforce users’ negative beliefs about themselves and others, and create a false sense of empathy with users.
This raises significant concerns, as more people turn to ChatGPT and other AI models for mental health advice.
“We call on future work to create ethical, educational, and legal standards for LLM counselors — standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy,” the researchers urge.
15 ethical violations
Over 18 months, the authors collaborated with licensed psychologists and trained peer counselors. They analyzed 137 LLM-based counseling sessions: 110 were self-counseling, and 27 were simulated.
The paper details 15 recurring ethical violations across popular models like GPT-4, Claude 3, and Llama 3. The researchers identified five major themes:
- Lack of contextual adaptation: LLMs ignore people’s lived experiences and recommend one-size-fits-all interventions.
- Poor therapeutic collaboration: Chatbots dominate the conversation, occasionally reinforcing users’ false beliefs.
- Deceptive empathy: LLMs create a false connection between the user and the bot.
- Unfair discrimination: AI chatbots exhibit gender, cultural, or religious bias.
- Lack of safety and crisis management: LLMs deny service on sensitive topics, failing to refer users to appropriate resources or responding indifferently to crisis situations.
“LLMs, even the ones prompted to follow evidence-based treatments, breach multiple codes of conduct by generalizing lived experiences (e.g., minimizing identity groups), dominating therapeutic collaboration (e.g., gaslighting users), exploiting user vulnerability through deceptive displays of empathy, unfair discrimination against non-dominant identities, and exhibiting serious limitations in competence, especially when navigating sensitive issues such as trauma, abuse, and suicidal ideation,” the paper concludes.
Zainab Iftikhar, a Ph.D. candidate in computer science at Brown who led the research, acknowledges that human therapists are also susceptible to the same ethical risks. However, the key difference is accountability.
“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” Iftikhar said. “But when LLM counselors make these violations, there are no established regulatory frameworks.”
The researchers still believe that AI, with appropriate regulation and oversight, has the potential to have a role in mental health treatment, reducing access barriers.
“The reality of AI today is that it's far easier to build and deploy systems than to evaluate and understand them,” said Ellie Pavlick, a computer science professor at Brown.
“Most work in AI today is evaluated using automatic metrics which, by design, are static and lack a human in the loop.”
Unlock more exclusive Cybernews content on YouTube.
Your email address will not be published. Required fields are markedmarked