OpenAI said it has axed clusters of accounts linked to state-backed threat actors, which were abusing ChatGPT to run influence campaigns and feed thousands of fake social media accounts that disseminated Russian and Chinese propaganda. Meanwhile, romance and recovery scammers were caught using ChatGPT accounts to generate fake identities, forge legal documents, and other materials.
The latest OpenAI report disclosing the disrupted malicious uses of ChatGPT confirms widespread use of AI models to power cybercrime and state-run influence operations.
The firm warns that threat actors are incorporating AI into their operations alongside more traditional tools such as websites and social media accounts.
“Threat activity is seldom limited to one platform,” OpenAI said.
“Rather, threat actors may use different AI models at various points in their operational workflow.“
The report focuses mostly on specific case studies and doesn’t assess the scale of the issue.
Some AI-generated social media posts gather tens of thousands of views
OpenAI said it had banned ChatGPT accounts linked to the Russian “Rybar” network, which disseminates generated content on a network of social media accounts.
“In some occasions, the threat actor used ChatGPT to generate batches of short social media comments, and these were then posted by accounts on X and Telegram that appeared to originate from different parts of the world,” the tech firm said.
The accounts primarily generated content for thousands of social media accounts. One user behind the content farm prompted ChatGPT to help draft covert interference campaigns in Africa.
“Users typically prompted in Russian, but generated content in a range of languages, notably Russian, English, and Spanish," OpenAI said in the report.
Many social media accounts had no declared connection to Rybar.
Some posts were highly successful. For example, one X account with 600,000 followers posted the generated tweet claiming that the West needs Moldova as a warzone, and it was viewed over 150,000 times.
“A separate prompt asked the model to edit a proposal for what appeared to be a deployed election interference team, apparently in Africa,” the report reads.
“This proposal included on-the-ground activity as well as online, such as building a network of local agents and organizing large-scale events. A third prompt discussed an information campaign focused on the Democratic Republic of Congo (DRC).”
Other prompts also asked about the electoral process in Burundi and Cameroon, and sketched out options for a campaign in Madagascar, including an estimated annual budget of up to $600,000 for one project.
Rybar network, sanctioned in some countries, has a large following across social media, with 1.4 million subscribers for its main Russian-language Telegram channel alone. Accounts on other platforms also have a significant following.
China's covert influence operations use ChatGPT
Another case study mentions banning a ChatGPT account belonging to an individual associated with Chinese law enforcement.
“The user’s activity revealed a well-resourced, meticulously-orchestrated strategy for covert IO against domestic and foreign adversaries, termed ‘cyber special operations.’ As part of this strategy, they tried to use our model to plan a covert IO targeting the Japanese prime minister, but our model refused,” OpenAI notes.
However, the smearing campaign went ahead anyway, using other tools. It unveiled a massive broader operation.
“The user’s engagement with ChatGPT included indications of much wider cross-internet activity, such as references to hashtags and fake accounts on social media. It also led to a website called revealscum[.]com, that we had already identified as part of the China-origin IO known as ‘Spamouflage’ in early 2024,” OpenAI said.
User prompts indicated that China runs “cyber special operations” to counter perceived hostile influence, they covered “analyzing and profiling targets; posting and amplifying content; working with online influencers; censoring unfavorable comments; and shaping the information landscape internationally.”
The user asked ChatGPT to help draft reports, indicating that 300 operators worked in their province alone and had been engaged in influence operations across hundreds of Chinese and “foreign” platforms. Other updates referred to equivalent teams in other provinces.
“The ChatGPT user’s reports included references to dissidents losing social media followers, reducing their activity, or even giving up entirely as a result of the harassment,” ChatGPT found.
A banned ChatGPT user recorded that their unit had made over 50,000 posts across more than 200 Western platforms.
“The user described millions of posts on Chinese networks and tens of thousands of posts on foreign ones, utilizing thousands of accounts, many of which were fake or working under the direction of the operation.”
Crooks automating romance, recovery scams
A few cases mention widespread scams originating from Cambodia, but led by Chinese users. One network of accounts was powering a semi-automated romance scam that likely defrauded hundreds of victims a month.
The scammers used ChatGPT to generate promotional texts for a high-end dating and escort service called “Klub Romantis.” The crooks then bought social media ads targeting rich Indonesian men interested in luxury lifestyle content, using keywords such as golf, yachts, and fine dining.
On Telegram, human operators were using API-powered ChatGPT automations to continue conversations, turning up the head and offering targets to join online dating platforms, showing generated profiles.
The “chosen girl” required increasingly larger payments for completing escalating “missions.” At the final stage, the scammers attempted to collect the largest payment, dubbed the “kill,” by inventing “compensation settlements,” “verification deposits,” or other fees.
Similarly, another Cambodia-linked operation had been abusing ChatGPT for a fake recovery scam, impersonating attorneys, fake law firms, and even the FBI to defraud previous scam victims. The scammers generated fake legal documents and other documents to sell false hope of “fund recovery.”
Many more case studies, shared by OpenAI, underscore “the importance of studying the nature of threat actors and the ways in which they behave, as well as the content they generate.”
Unlock more exclusive Cybernews content on YouTube.
Your email address will not be published. Required fields are markedmarked