Sexualized surveillance: experts warn against using ChatGPT erotica

Chatbots allowing users to engage in raunchy conversations can be a powerful tool of surveillance, comparable to espionage through intimate partners, an expert says.

OpenAI will soon roll out a new version of ChatGPT, providing erotica for age-verified users as part of the company’s “treat adult users like adults” principle, CEO Sam Altman announced mid-October.

The news was met with backlash, as critics cited flaws in age verification systems, potentially allowing children to access pornography. Not to mention the societal implications of forming intimate relationships with chatbots.

Reacting to mounting criticism, Altman doubled down by saying that OpenAI isn’t “the elected moral police of the world,” and compared the chatbot’s erotica with R-rated movies.

Ok this tweet about upcoming changes to ChatGPT blew up on the erotica point much more than I thought it was going to! It was meant to be just one example of us allowing more user freedom for adults. Here is an effort to better communicate it:

As we have said earlier, we are… https://t.co/OUVfevokHE

undefined Sam Altman (@sama) October 15, 2025

Some artificial intelligence (AI) security experts say the end goal of tech companies that own sexually explicit chatbots may be surveillance.

According to Disesdi Susanna Cox, an AI architect and security researcher, tech companies cannot build artificial general intelligence, which is the type of AI that surpasses human intelligence capabilities. Nor can they make current large language models (LLMs) smarter.

“The only really viable profit path for this technology is surveillance,” Cox tells Cybernews.

Making chatbots addictive

By allowing erotica, the system will have almost everything it needs to be fully addictive, Cox says. It will not only maximize users’ time spent on the platform but also help build a sense of trust due to the perceived intimate nature of the conversation.

“Through the history of espionage, one of the most effective means of intelligence gathering is through intimate partners. From my perspective, this becomes a new, incredibly easy tool for mass surveillance,” she says.

The data users provide to chatbots like ChatGPT – including sexual conversations – can be used to train LLMs.

Vasant Dhar, a professor at NYU’s Stern School of Business and the Center for Data Science, says the data is needed to learn how to make money in the best possible way as quickly as possible.

“Or, to keep you online, to hold your attention, which leads to monetization. At the end of the day, AI is a commercial activity, and these companies are in it to make money,” he says.

Another potential use of your data is targeted advertising. Jeremy Blackburn, an associate professor at the Binghamton University School of Computing, says OpenAI is already working on an ad platform where having more data is an advantage.

“It’s a bit dystopian, but there are lots of relevant data points for advertisers that can be extracted from sexual conversations. Anything from a relatively mundane preference of red over white wine to straight-up kink-derived preferences like leather clothing,” Blackburn tells Cybernews.

Through the history of espionage, one of the most effective means of intelligence gathering is through intimate partners. From my perspective, this becomes a new, incredibly easy tool for mass surveillance,

Disesdi Susanna Cox

Cox says using erotic conversations for personalized ads is the best-case scenario of how data could be used. In a more dire scenario, the chatbot can act as a honeypot, which is especially concerning because tech companies share their data with government agencies.

“Once they have this data on you, they are able to build very in-depth psychological profiles,” she says.

According to OpenAI, users who express suicidal intent or plan to harm others can be referred to a crisis hotline or law enforcement.

Such safeguards could potentially help prevent more tragic cases like the suicide of a 16-year-old Adam Raine, who took his own life after confiding to ChatGPT. Some vulnerable groups, however, may be wary of law enforcement accessing their sensitive data.

“Should we ever reach a point where being LGBTQ+ is criminalized in any way – as is already the case in several countries around the world, your LLM transcripts may be turned over for possible prosecution,” Bo Young Lee, a CEO of non-profit AI4ALL, wrote on LinkedIn.

Some countries where ChatGPT is supported, like Libya and Saudi Arabia, criminalize same-sex sexual activity. It’s unclear if OpenAI would ever share such data with these governments, given that Altman advocated for the safety of the queer community.

Conversations may become public

Even if deepest sexual fantasies confided to a chatbot are not shared with autocratic governments, they can still end up in the wrong hands.

Blackburn says a typical data breach could cause users many problems. The breach of Ashley Madison, a dating website for married people who wish to cheat on their spouse, didn’t include detailed chat logs, but many people’s lives were still affected.

It’s a bit dystopian, but there are lots of relevant data points for advertisers that can be extracted from sexual conversations. Anything from a relatively mundane preference of red over white wine to straight-up kink-derived preferences like leather clothing.

Jeremy Blackburn

“Considering that there have been multiple instances of ChatGPT leaking private information that had been a part of its training data, this means there is a very real risk that users’ particular sexual preferences will be encoded into and revealed by a future model,” Blackburn says.

Age verification, aimed at ensuring that only adult users will access erotica, also poses security risks, according to Yaron Litwin, online safety expert and CMO of the Canopy Parental Control App.

He says ChatGPT’s age verification can require users to upload a picture of themselves or a government-issued ID.

Litwin tells Cybernews, “In theory, erotic chat content could be linked to such identifying information.”

Unlock more exclusive Cybernews content on YouTube.