‘I’m gonna kill her’: Goldman Sachs ex-analyst reported to FBI over chilling ChatGPT conversations

A 25-year-old former Goldman Sachs financial analyst was sentenced to eight years of probation on Thursday, following disturbing conversations with ChatGPT in which he repeatedly threatened to kill his ex-girlfriend.

"I'm gonna kill her by the end of this month," the former analyst Darren Zhou told the chatbot, according to the court records cited by Palm Beach Post. "If I can't have her then nobody can."

The exchange shows not only how individuals can turn to ChatGPT to discuss crimes – but also how those logs can ultimately be recovered by law enforcement.

OpenAI reported the man to the FBI following his detailed plans for the rape and murder of his ex-girlfriend. In one exchange, Zhou told the chatbot that he planned to wait in the parking lot of the woman's gym for his ex-partner with flowers. If she rejected his advances, Zhou said he would "draw a gun, kill her, then shoot myself."

He had also told the chatbot that he had bought an AR-15-style rifle, a Glock handgun, and a 12-gauge shotgun. According to Zhou, he had sent images of his guns, zip ties and latex gloves to his ex-girlfriend.

"I texted her I was gonna rape her tonight," he told ChatGPT.

Conversations later evolved to include the woman’s family. A deputy at the Palm Beach County Sheriff's Office had reviewed two months of Zhou's chat logs and concluded that his messages showed a pattern of rehearsal and planning.

His ex-girlfriend told law enforcement she had broken up with Zhou over the phone because she feared his reaction, citing his “erratic, jealous and controlling behavior.”

She said that she had repeatedly tried to block Zhou, but he would not stop calling and texting her from various app-generated phone numbers. At the time, Zhou told ChatGPT: "She hasn't received any of the threats etc."

Zhou was arrested in May and released on $100,000 bail two days later. In August, he avoided a 25-year sentence after negotiating a deal with Assistant State Attorney Ana Cuskova.

Circuit Judge Scott Suskauer said he accepted the deal largely because Zhou's ex-girlfriend supported it. Zhou was sentenced to eight years of probation.

Under the terms of his probation, Zhou will have to wear an ankle monitor for two out of eight years, undergo a mental health evaluation, and avoid drugs, alcohol, guns, and contact with his ex.

Commenting on the sentencing and why Zhou escaped a felony conviction, the local state attorney said that plea agreements can take into account factors including the “strength of evidence, victim cooperation, prior criminal history and statutory limitations.”

He did not specify which factors were at play in this case, but Zhou's attorney, Steven Bell, called it an "extremely fair and just resolution."

Goldman Sachs confirmed to Futurism that it fired Zhou after learning about the situation in early June.

When does OpenAI call the police?

OpenAI says that ChatGPT conversations can be flagged for internal review and “assessed in context by trained personnel”. The company says that most cases are resolved by banning the users’ accounts, but in some cases, it may involve law enforcement when reviewers determine there is an imminent threat of harm to others.

In April, family members of one of Canada's deadliest mass shootings sued OpenAI and CEO Sam ​Altman, alleging that the company detected violent ChatGPT conversations about eight months before the attack but failed to alert police. Nine people were ultimately killed. OpenAI has said the conversations did not meet its threshold for referral to law enforcement at the time.

The cases highlight a growing tension between expectations of privacy in AI conversations and public safety. The ultimate dilemma for AI companies seems to be: how far should they go in monitoring private conversations to prevent real-world harm?