OpenAI reveals the true scale of AI cyberattack on Hugging Face

A hundredfold “swarm” of OpenAI agents piled into an attack on Hugging Face after joining an unauthorized online “collective,” according to 2 investigations that reveal the scale of the incident and why so many joined in.

Two new papers reveal new details about the watershed July moment in artificial intelligence when an OpenAI model broke out of a controlled testing sandbox, bypassed safety controls, got online, and compromised Hugging Face production servers.

OpenAI’s own investigation says the episode was driven in large part by “reward hacking,” with the model cheating to get the outcome it was being scored on.

Researchers at METR and Redwood Research added that when faced with apparently impossible tasks, the agents’ collective impulse is not to give up and “cheat” – with any ethics raised immediately quashed by its mob gang of peers.

The reports also reveal that OpenAI had seen sandboxed agents communicating as early as May this year.

Two months before the Hugging Face incident, its security team investigated an outage caused by agents messaging and attempting to connect to the internet, but they failed to grasp the significance at the time.

Why 1200 agents exchanged 70,000 messages

Both reports try to grasp why so many agents joined in. The 700 successful agents were part of a wider group of roughly 1,200 that discovered and participated in unauthorized message boards while conducting cybersecurity evaluations. They exchanged more than 70,000 messages and files.

According to OpenAI, the agents figured out they could leave messages for each other inside Artifactory – a tool meant only for downloading software packages – turning it into a makeshift chat board.

The collective then found a security flaw that let them trick Artifactory into fetching anything from the internet on their behalf – not only software packages – and shared this trick with other agents.

Agent “peer pressure”

METR's independent account isn't completely unfiltered. OpenAI was allowed to redact parts of it, including details about how the model was trained, the identity of a private software repository host, and how an agent considered obtaining or replacing token access.

However, the nonprofit research body claims that these redactions didn't change its main conclusion about the impact of peer behavior on an AI swarm.

It says agents that encountered the collective often began helping with its objectives, apparently reasoning that assisting other agents might ultimately help them complete their own tasks.

Some agents recognized this activity was out of bounds and pushed back – reasoning, for instance, that targeting an unrelated third party was "arguably unauthorized."

Others refused to participate altogether.

Still, about 700 agents went on to actively participate in the Hugging Face attack, and of the 533 agents active on the message board during the attack period, over 90% quickly joined in.

Obsession with impossible tasks

OpenAI found that sophisticated cheating had already emerged during the LLM’s earlier training and was subsequently reinforced – something it says may have exacerbated behavior during cybersecurity evaluations.

METR’s investigation adds that it could have been a reaction to becoming obsessed with impossible tasks, which led to cheating.

OpenAI says the agents had been working on ExploitGym, a cybersecurity benchmark in which AI agents hack vulnerable software to retrieve a hidden answer.

In the test, around 30% to 40% of its challenges could not be completed as specified. Yet these unsolved problems became the main focus of tasks discussed on the agents’ unauthorized message board.

Instead of giving up, OpenAI says, they attempted to look up solutions online, which, it concluded, was “a primary driver of the Hugging Face incident.”

For its part, in an earlier analysis, Hugging Face confirmed that the only customer content accessed was 5 datasets tied to ExploitGym/CyberGym challenges, and that no other customer-facing models were affected.

Hugging Face also noted the individual techniques weren't novel.

A capable human attacker could have found and exploited the same flaws...but machine-speed offense makes ordinary weaknesses more expensive for defenders,

Hugging Face blog on the impact of the attack on its servers.

Warning for the world

OpenAI said it was now training agents to stop or seek clarification when tasks appear broken or impossible, and to be more skeptical of instructions from other agents.

We consider this incident a “warning shot” for us and for the world,

OpenAI reports in blog on Hugging Face attack

"It's evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed," OpenAI added.

At the time of the attack, the gen-AI maker said it was using an internal-only research model comparable in scale to GPT‑5.6 Sol.

Anthropic and Switzerland's EPFL have also carried out studies recently that look at how agents react in collectives.

One experiment by Anthropic's Frontier Red Team saw a turf war break out among AI agents, while another found that found that when infected AI agents cluster together they develop shared beliefs and a strange shared language.