Foreign adversaries are now building AI into their existing workflows – from crafting phishing campaigns, tweaking malware, and generating propaganda, to researching ways to automate their cyber kill chain, according to a new report by OpenAI.
-
A new OpenAI report finds foreign threat actors are abusing ChatGPT and other AI tools to enhance phishing, malware, and propaganda.
-
Most threat actors use AI tools with previously known TTPs, hoping to boost speed and efficiency, rather than develop new attack capabilities.
-
Since OpenAI began its public threat reporting, over 40 networks have been disrupted for violating its AI model usage policies.
But, in what may be considered good news for security teams, the AI start-up also says most threat actors appear to be playing it safe and sticking with "tried and true" methods previously used to carry out their attacks.
“We continue to see threat actors bolt AI onto old playbooks to move faster, not gain novel offensive capability from our models,” the company said in a security blog published on Tuesday.
The latest observations are documented in OpenAI’s newly released 3rd quarter security report, “Disrupting malicious uses of AI: an update.”
The 37-page report is chock full of the company’s most recent run-ins with various nation-state actors abusing ChatGPT and multiple other AI tools, including those from DeepSeek and Anthropic, for nefarious purposes.
Since February 2024, OpenAI has been monitoring the use of its AI tools using common sense rules that “protect people from actual harms, and build democratic AI,” it said.
This includes actively blocking the nefarious use of AI tools to protect society at scale from “malicious cyber activity, organized crime and scams, and covert influence operations (IO),” and actively preventing authoritarian regimes from “using AI to amass power, control citizens, or threaten other countries.”
OpenAI says since it began the public threat reporting, it has uncovered and disrupted over 40 networks violating its usage policies.
Threat actors play it safe
Laid out in case study style, OpenAI says the biggest takeaway from the security report is that “Repeatedly, and across different types of operations, the threat actors we banned were building AI into their existing workflows, rather than building new workflows around AI.”
The report profiles typical nation-state actors, including those from Russia, North Korea, and the People’s Republic of China (PRC), as well as scam networks located in Cambodia, Myanmar, and Nigeria.
Overall, OpenAI said it banned several Russian-speaking criminal accounts attempting to use GPT models “to help develop and refine malware, including a remote‑access trojan, credential stealers, and features to evade detection,” more aligned with refining offensive tooling rather than executing them.
One Russian malware developer, discovered vibecoding remote access tool functionalities, was also discovered using multiple ChatGPT accounts to prototype and troubleshoot technical components to enable post‑exploitation and credential theft.
"We found no evidence that access to our models provided these actors with novel capabilities or directions that they could not otherwise have obtained from multiple publicly available resources,' OpenAI said.
Besides malware, one case featured Korean operators attempting to use LLM models for command-and-control (C2) development, cryptocurrency-themed phishing content, experimenting with HTML obfuscation, and proxying reCAPTCHA for convincing login pages.
A cluster of Chinese-language ChatGPT accounts, run by an actor deemed technically competent but unsophisticated, was allegedly used to send phishing messages after creating “detailed and formulaic” phishing content.
That same PRC actor also attempted to use OpenAI models “to plan and iterate on encrypted C2 components, remote code execution, reconnaissance/process control, and to gain how-to help for installing and using open source tools.
Scammers, propaganda, and government abuse
OpenAI says all of the scam operations identified and banned this year "primarily used AI as a scaling and efficiecy tool."
When it came to authoritarian abuses of AI, OpenAI discovered China-linked accounts asking GPT models “to generate work proposals for large-scale systems designed to monitor social media conversations.”
Beijing-linked accounts were also found using the AI service for targeted research and profiling of so-called Chinese dissidents.
In another case tied to a previously seen Russian influence campaign called “Stop News,” ChatGPT was asked to create content for social media posts, fake foreign news sites, and to generate scripts for short news-style videos.
Highlighting the prevalence of these types of operations, last month, an ex-cop in Florida was busted for running a Russian influence campaign dubbed “CopyCop” out of his house, consisting of hundreds of fake Kremlin propaganda websites created by Meta’s Llama 3.
One interesting observation was that OpenAI witnessed several savvy threat actors becoming wise to the use of em-dashes in ChatGPT-generated text, purposefully removing the dashes before attempting to publish their text.
From lone actors to seasoned crime cartels, foreign scam networks were also prevalent adopters of AI tools, often with the end goal of committing financial fraud.
Posing as trading experts or job recruiters to lure victims into private messaging groups, scam actors in Nigeria and Cambodia were found using AI to create fake social media accounts, fake ads, and websites for non-existent companies.
Although some AI use was elaborate, OpenAI says the majority of scammer interactions with ChatGPT featured relatively simple tasks like translation, proving, once again, the criminals' "playbook" stayed the same.
Furthermore, OpenAI noted a simultaneous rise in ChatGPT being asked by users to identify if content was AI-generated by scam artists, preventing fraud and deception.
So what is OpenAI doing to help reduce AI interference when it comes to nation-state adversaries?
Curious what others think about this story? Contribute your thoughts to the debate below.
“When activity violates our policies, we ban accounts and, where appropriate, share insights with partners,” OpenAI said, aiming to raise awareness of abuse and improve protections for everyday users.
“We are dedicated to identifying, preventing, and disrupting attempts to abuse our models for harmful ends,” the company states.
Your email address will not be published. Required fields are markedmarked