In much of the world, demand for mental health services usually far outstrips the supply of support, so it's perhaps no great surprise that so many of us are turning to large language models, like ChatGPT, for mental health support. The question is: is this a good thing? There have already been examples of people harming themselves, or even committing suicide, after conversations with their chatbot.
Research from Brown University shows how dangerous this is and how roughshod most chatbots are when it comes to basic ethical standards. The study compared how chatbots operate with standards from bodies such as the American Psychological Association, and found the bots to be severely lacking.
Breaking the rules
The study found that chatbots respond in a number of ways that aren't recommended by official bodies. For instance, they might provide responses that reinforce the negative beliefs users have about themselves, or create a false sense of empathy with users.
“In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice by mapping the model’s behavior to specific ethical violations,” the researchers explain.
“We call on future work to create ethical, educational, and legal standards for LLM counselors — standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy.”
Mental health support
According to research, around half of users with a mental health condition use chatbots for support. The Brown researchers wanted to test how helpful that was. They input a wide range of different prompts to explore the kind of responses chatbots typically provide.
Rather than aiming to trip up the bots, the researchers accept that users are turning to AI for support and want to see how the bots can do better. The aim was to see if different prompts provide better answers.
“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task,” the researchers explain. “You don’t change the underlying model or provide new data, but the prompt helps guide the model’s output based on its pre-existing knowledge and learned patterns."
The conversation on this topic is live. Join in the discussion.
This is obviously slightly easier to achieve than changing either the data or the model. It requires users to actively frame the conversation in a professional way, such as asking the bot to behave as a cognitive behavioral therapist when reframing thoughts.
"While these models do not actually perform these therapeutic techniques like a human would, they rather use their learned patterns to generate responses that align with the concepts of CBT or DBT based on the input prompt provided," the researchers continue.
Smarter responses
The researchers note that individual users are already experimenting with prompts to guide chatbots toward more therapeutic responses. Many share their preferred phrasing on social media platforms such as TikTok, Instagram, and Reddit, where communities exchange advice on how to elicit more empathetic or helpful replies. But the implications of prompting go far beyond casual experimentation.
A growing number of mental health apps are built on top of general-purpose large language models that have simply been “steered” with mental health–related prompts. Understanding how these prompts shape the behavior of such systems is therefore essential if they are to be used safely and effectively.
To explore this, the researchers observed a group of peer counselors working with an online mental health support platform. Seven counselors, all trained in cognitive behavioral therapy techniques, conducted self-counseling sessions with LLMs prompted to emulate CBT practitioners.
Not meeting the grade
A smaller subset of sessions was then assessed by a group of trained clinical psychologists. They tried to compare the exchanges people had with AI against the standards and ethics they adhere to in their professional practice. In total, this analysis revealed 15 ethical risks that fell into five broad categories:
- Lack of context, with the AI often applying a one-size-fits-all approach and not taking into account the users' particular circumstances.
- A one-way dialog, with the AI seldom engaging in a more collaborative approach to problem-solving. The AI would also often reinforce harmful self-beliefs that the user had.
- The faux empathy AI exhibits has been well documented, and this was evident in the research, with the chatbot trying to simulate a genuine connection.
- The chatbot also showed clear gender, cultural, and religious biases, which made its responses discriminatory.
- Last, but by no means least, it struggled to respond effectively to crisis situations, such as suicidal ideation.
Suffice it to say, human therapists are by no means perfect either, but the scale of adoption of generative AI for mental health-related conversations makes this an area that needs to be urgently addressed, not least because professional therapists are accountable in a way that the AI companies are, thus far at least, not.
The researchers emphasize that their findings should not be interpreted as an argument against the use of AI in mental health support. Instead, they see the technology’s potential to expand access to care — especially where cost or availability remain barriers. What’s needed, they argue, is thoughtful design, robust oversight, and clear ethical standards to ensure AI systems do more good than harm.
“There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good,” the researchers conclude. “This work offers a good example of what that can look like.”
Unlock more exclusive Cybernews content on YouTube.
Your email address will not be published. Required fields are markedmarked