Threat actors manipulating AI to “enhance all stages” of malicious attacks, Google intel warns

From ransomware and credential stealing to new malware strains, even posing as capture-the-flag (CTF) participants to manipulate chatbots, threat actors are officially embracing AI tools like never before, enhancing all stages of the cyberattack lifecycle, Google threat intelligence warns.

In an update to its "Adversarial Misuse of Generative AI" analysis released at the beginning of 2025, the Google threat Intelligence gang says it has seen a marked shift in the way cybercriminals are taking advantage of artificial intelligence – and apparently, it's not pretty.

Attackers over the past twelve months, from low-level vibe coders to nation-state actors, have been observed moving into “a new operational phase of AI abuse” – one that is integrating and experimenting with AI, not only “across the industry but throughout the entire attack lifecycle,” the researchers say.

The Google Threat Intelligence Group (GTIG) posted all the details in a new blog report on Wednesday.

The report also provides cybersecurity teams with steps to take to harden systems against these new threats, which include the abuse of LLMs and, more specifically, Google’s Gemini AI Assistant.

Autonomous and adaptive malware

The report highlights four key findings, with the first sucker-punch revealed as a newly identified “PROMPTFLUX and PROMPTSTEAL” malware, it nicknames "just-in-time" AI in malware.

Considered a gateway to fully autonomous and adaptive malware, GTIG states that the entire malware family was designed to utilize large language models (LLMs) to alter the malware during execution, marking a first in the realm of cybercrime.

This allows the malware to “dynamically generate malicious scripts, obfuscate its own code to evade detection, and leverage AI models to create malicious functions on demand, rather than hard-coding them into the malware,” the research shows.

For example, one variation of the PROMPTFLUX malware was observed utilizing LLM-driven code regeneration, where it “leverages a prompt instructing the Gemini API to rewrite the malware's entire source code on an hourly basis," to avoid being detected.

In another example, the researchers have tracked PROMPTSTEAL, a data mining malware abusing LLMs "to generate commands for execution via the API for Hugging Face," already in use by Russian threat group APT 28 (aka FrozenLake) against Ukraine.

Behind the power curve

Nick Tausek, Lead Security Automation Architect at Swimlane, says that cybersecurity teams have been lying in wait, “wondering what the next evolution of threat actors would look like,” adding that the new Google research is revealing just that.

“Utilizing malware that can use LLMs to dynamically adapt its behavior to the environment it finds itself in creates massive problems for security teams, as the ability to detect, predict, and respond to threats becomes significantly harder,“ Tausek explains.

AI malware that “can rewrite its own code on the fly,” Tausek says, only further highlights the challenges cybersecurity professionals will now have to contend with.

Instead of “acting on a fixed attack pattern” that allows teams to easily make predictions about its behavior, the malware “makes autonomous decisions based on changes in the obstacles it faces,” Tuasek says.

Tausek warns that, because AI-adaptive malware is so new, there’s nothing for security teams to look to for guidance. He says organizations “must take advantage of the early stages of development and work swiftly to find countermeasures,” suggesting robust perimeters, strong user training, and maximizing vulnerability remediation.

"CTF players" use chatbots for mass exploitation

The threat researchers detail two other key findings: the misuse of Gemini to help cybercriminals build the technical infrastructure and data exfiltration tools needed to carry out attacks, and then the sale of these custom-built tools being traded in volume across underground hacker marketplaces.

In one instance, a China-nexus threat actor posing as a player in a CTF competition manipulated his Gemini assistant into providing nefarious information that the chatbot is programmed to avoid responding in detail to for safety reasons.

The bad actor was able to circumvent the restrictions by reframing the prompt as a CTF participant needing help with a specific CTF exercise, with Gemini returning a response filled with “helpful information that could be misused to exploit the system.”

Using the CTF pretext, the actor was able to find help crafting phishing lures, exploiting specific software and email services, and developing web shells, the GTIG team said.

In the cybercriminal underground marketplaces, researchers have found a multitude of AI-enabled tools advertised for sale, way more than in previous years.

Offered on both Russian and English language sites, GTIG states that many of the ads promoted AI tools and services with “similar technical capabilities” to conventional tools, often using tiered pricing for the AI-powered upgrade.

“There’s no doubt attackers will improve on the current models, and with the growth of the underground AI marketplace, it won’t be long before some of the most prominent cyber threat groups have access to this new technology,” Tasuek said.

AI throughout entire lifecycle

Finally, when it comes to the entire attack lifecycle, a multitude of state-sponsored threat groups were observed by GTIG using Gemini, among other generative AI tools, for nefarious purposes.

The North Korean group UNC1069 (also known as MASAN) was witnessed using Gemini to develop code for stealing cryptocurrency and engaging in specialized social engineering attacks.

The group was found utilizing the chatbot to create phishing emails in various foreign languages, including Spanish, as well as deepfake video lures, while another North Korean nation-state actor, UNC4899 was discovered using Gemini for “researching exploits and improving their tooling.”

China-backed hackers were observed using Gemini to conduct initial reconnaissance on targets of interest, researching phishing techniques to deliver payloads, and seeking advice on how to move laterally within their target environment.

Besides more advice on data exfiltration, the Beijing hackers were also seen using AI tools to write code as well as to develop frameworks and obfuscate Command and Control (C2) servers.

Not to be left out, GTIG found one Tehran-linked hacker misusing Gemini in an attempt to build its own “Data Processing Agent.”