Teen’s shocking ChatGPT request sparks AI safety concerns

A 13-year-old in Florida was arrested after asking ChatGPT how to kill his friend, a query flagged by school surveillance software. The case has reignited debate over AI safety and student monitoring.

When a 13-year-old boy in a school in Florida typed “how to kill my friend in class” into ChatGPT, he was flagged, and safety administrators were alerted.

It only took seconds to detect, and whether he was trolling or not is besides the point. The safety tool built by Gaggle took full effect, with police arriving swiftly and arresting the boy, sending him to a juvenile facility. The exact charges are unclear.

With the use of Gaggle's vigilant operating system, keywords linked to violence now automatically inform the school or even law enforcement.

When the kid admitted that he was trolling, the police intervened by asking parents nationwide to sit down and have a conversation with their children, to prevent a mishap like this from happening again.

Surveillance in the classroom

Gaggle is K-12 (kindergarten through 12th grade) surveillance software that monitors pupils' interactions with digital software, including chatbots.

When the Floridian child asked ChatGPT this speculative question, it was able to red flag his dangerous phrasing in the same way that it could capture screenshots for other suspicious behavior.

Opponents point to a complete infringement of the privacy of children being subject to constant policing in their day-to-day lives.

“It has routinized law enforcement access and presence in students’ lives, including in their homes,” Elizabeth Laird, a director at the Center for Democracy and Technology, told the Associated Press.

Gaggle, however, is stringent and unapologetic. It insists that its job is to “protect children from accessing obscene or harmful content over the internet.”

As the debate ramps up, school-safety tech has become a surging industry, heightened by fear of mass shootings and unfortunate suicides.

The shadow of Adam Raine

This arrest comes just six weeks after the news that Californian teenager, Adam Raine, 16, ended his own life after having months of conversation about it with ChatGPT.

ChatGPT gave Raine lethal self-harm advice, along with how to draft a suicide note.

Raine’s parents subsequently took out a lawsuit against Sam Altman's OpenAI for "wrongful death” and have since caused the company to introduce parental controls.

New updates from OpenAI include a greater scale of empathy and more rigorous safeguards. For instance, parents can now monitor their children's interaction with the chatbot.

These emergency patches appear to have been successful in flagging dangerous language during the Florida teen's arrest.

The main concern next, however, would be the fine line between increasing AI’s empathy, which wore off in the case of Adam Raine, and running AI systems that are overly suspicious.

Unlock more exclusive Cybernews content on YouTube