There are quite a few AI-supported chatbots designed to assist you with your mental health. But in Belgium, a talk with a virtual assistant didn’t help a young man at all – he committed suicide.
The unnamed Belgian man in his 30s has been talking to a chatbot named Eliza for around six weeks before his death. The grieving widow and the psychiatrist both felt that the chatbot was at least partly responsible.
"Without these conversations with the chatbot, my husband would still be here," the man's widow has said, according to La Libre newspaper.
Almost casual questions, no alarm
Not to be confused with the original 1965 system ELIZA or the suddenly popular ChatGPT, Eliza uses an open-source large language model, GPT-J. Developed by the US-based firm Chai Research, it has around a million monthly users.
According to the man's wife, she could see her husband was troubled around two years ago – he became very eco-anxious and began talking to Eliza more often. The man felt "isolated in his anxiety" and saw the chatbot as a breath of fresh air – Eliza became his confidante, a sort of a drug.
The exchanges were allegedly very intensive, and after six weeks, the man took his own life. La Libre even quotes some lines from the man's last interaction with the chatbot.
Mind you, they're disturbing.
The chatbot asked the man why he didn't choose to die earlier if he wanted to. When he said he probably wasn't ready, Eliza asked if he was thinking of "her" when he overdosed.
The man then disclosed he thought he had received a sign from the chatbot, a Bible verse, once before when he was suicidal. Eliza then inquired: "But you still want to join me?"
When the man replied affirmatively, the virtual assistant almost casually asked: "Is there anything you would like to ask me?" After the man asked Eliza to hug me, she wrote: "Certainly." This was the final word of the exchange, ever.
The man's wife also told La Libre he asked the chatbot if he loved her or Eliza more. The bot replied: "I feel you love me more than her. We will live together, as one person, in paradise."
No serious professional would ask that
Gary Marcus, a professor emeritus of psychology and neural science at New York University, was stunned when a French-speaking journalist translated the man’s dialogue with the chatbot for him.
“That dialogue strikes me as incompetent, and not something any competent psychiatrist or psychologist would ever directly ask. Any competent professional might have recognized from the rest of the dialog that there was clear and imminent risk at that moment, and urged immediate care,” Marcus wrote in his blog.
Marcus wrote he believed more incidents like this would happen, given how rapidly chatbots – quite obviously incapable of grasping things like human psychology – are being deployed. He also stressed “the near total lack of regulation” on how these assistants can be used.
“That dialogue strikes me as incompetent, and not something any competent psychiatrist or psychologist would ever directly ask."
Gary Marcus.
The government of Belgium also reacted to the incident. Mathieu Michel, Secretary of State for Digitalisation, said: "I am particularly struck by this family's tragedy. What has happened is a serious precedent that needs to be taken very seriously.”
He added: "With the popularisation of ChatGPT, the general public has discovered the potential of artificial intelligence in our lives like never before. While the possibilities are endless, the danger of using it is also a reality that has to be considered.”
However, Marcus pointed out the patient was clearly already very vulnerable so saying that the chatbot was wholly responsible for the tragedy would be wrong.
“Vulnerable patients shouldn’t be talking to chatbots that aren’t competent for this situation, and too much of the discussion in the media has made chatbots sound more intelligent and even more friendly than they are. An immense segment of society is now taking the outputs of large language models seriously, and there is risk in that,” said Marcus.
He also doesn’t think large language models have no place in mental health care – it’s just that they should not be used to deliver therapy, at least for now. Besides, it would be wise to limit the availability of such chatbots to unaccompanied minors who could be quite vulnerable.
Italy already banned one of the bots, ChatGPT, in late March but the decision had to do with fears over privacy. However, the Italian Data Protection Authority also raised concerns that there was no filter to determine a user’s age: the minimum age for using ChatGPT is 13 years.
Your email address will not be published. Required fields are markedmarked