People who show violent extremist tendencies on ChatGPT will be directed to human and chatbot‑based deradicalization support through a new tool in development in New Zealand.

The initiative is the latest attempt to address safety concerns in the face of a growing number of lawsuits accusing AI companies of failing to stop, and even enabling, violence.

OpenAI was threatened with intervention by the Canadian government in February after revealing that 18-year-old Jesse Van Rootselaar, who carried out a deadly school shooting, had been banned by the platform without authorities being informed.

Authorities say that during the attack at Tumbler Ridge Secondary School, 9 people, including the attacker, died. Dozens more were injured.

ThroughLine, a startup hired in recent years by OpenAI, Anthropic, and Google, redirects users to crisis support when they're flagged as being at risk of self-harm, domestic violence, or an eating disorder. The company is now exploring ways to expand its services to include preventing violent extremism, its founder and former youth worker Elliot Taylor said.

The company is in discussions with The Christchurch Call, an initiative to stamp out online hate formed after New Zealand's worst terrorist attack in 2019. The partnership would involve the anti-extremism group providing guidance while ThroughLine develops the intervention chatbot, Taylor said.

"It's something that we'd like to move toward and to do a better job of covering and then to be able to better support platforms," Taylor said in an interview, adding that no timeframe has been set.

OpenAI confirmed the relationship with ThroughLine but declined to comment further. Anthropic and Google did not immediately respond to requests for comment.

Taylor's firm, which he runs from his home in rural New Zealand, has become a go-to for AI firms with its offer of a constantly-checked network of 1,600 helplines in 180 countries.

Once the AI detects signs of a potential mental health crisis, it routes the user to ThroughLine, which matches them with an available human-run service nearby.

However, ThroughLine's scope has been limited to specific categories, the founder said. The breadth of mental health struggles that people disclose online has exploded with the popularity of AI chatbots, and now includes dalliances with extremism, he added.

ChatGPT has been at the center of legal action in the US after the parent of a deceased 16-year-old had sued Sam Altman, the founder of the company. The family alleges the teen spent months discussing his suicide with the bot, which gave tips on self-harm methods and encouraged the idea.

Here’s how the deradicalization tool would work

The anti-extremism tool would probably be a hybrid model combining a chatbot trained to respond to people who show signs of extremism and referrals to real-world mental health services, Taylor said.

"We're not using the training data of a base LLM," he said, referring to the generic datasets large language model platforms use to form coherent text. "We're working with the correct experts."

The technology is currently being tested, but no date has been set for release.

Galen Lamphere-Englund, a counterterrorism adviser representing The Christchurch Call, said he hoped to roll the product out for moderators of gaming forums and for parents and caregivers who want to weed out extremism online.

A chatbot rerouting tool was "a good and necessary idea because it recognizes that it's not just content that is the problem, but relationship dynamics," said Henry Fraser, an AI researcher at Queensland University of Technology.

The product's success may depend on questions of "how good are follow-up mechanisms and how good are the structures and relationships that they direct people into at addressing the problem," he said.

Taylor said follow-up features, including possible alerts to authorities about dangerous users, were still to be determined but would take into account any risk of triggering escalated behavior.

He said people in distress tended to share things online that they were too embarrassed to say to a person, and governments risked compounding danger if they pressured platforms to cut off users who engaged in sensitive conversations.

The Cybernews community is talking about this. Be a part of the conversation.

Heightened moderation associated with militancy by platforms under pressure from law enforcement has seen sympathizers moving to less regulated alternatives like Telegram, according to a 2025 study by New York University's Stern Center for Business and Human Rights.

"If you talk to an AI and disclose the crisis and it shuts down the conversation, no one knows that happened, and that person might still be without support," Taylor said.

---

Unlock more exclusive Cybernews content on YouTube.