OpenAI agents attacked RubyGems months before rogue Hugging Face hack

Autonomous AI agents being tested by OpenAI flooded the popular RubyGems software repository with thousands of malicious packages roughly two months before hacking Hugging Face, researchers say.

According to the underlying investigation by researchers Spencer Kitts, Thomas Larsen, and Sydney Von Arx, the agents published malicious packages and attempted to steal RubyGems user API keys by exploiting a previously unknown vulnerability in the RubyGems server.

The first package was uploaded on May 5th, and between May 11th and May 12th, more than 2,000 packages were dumped on RubyGems.

According to the findings, the agents “clearly regarded what they were doing as hacking”, using file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. Additionally, the campaign was filled with comments like “#malicious probe” or “#hack”.

Multiple packages were found to disarm themselves to hide their payload in the next version. In one instance, a package contained the comment “#disable evil in next version and bump version,” followed by instructions to remove the malicious code in its next version.

RubyGems initially described the traffic as an ongoing DDoS attack and disabled new user registrations. It later blocked and removed the accounts responsible and yanked more than 500 malicious packages.

“Gem installs and pushes for existing users remained unaffected, and registrations reopened on May 16,” RubyGems said.

The researchers say they could not determine whether the attack was successful, while RubyGems said it found “no evidence” that attempts to obtain users' API keys succeeded.

Researchers attributed the activity to OpenAI agents based on the LLM origin of the packages, agents identifying themselves as OpenAI systems, and “extreme” similarities to agents involved in the German wiki incident.

Hundreds of the uploaded packages contained “oai” in their names, while 15 set “oai” as their author. One also listed an email for contact as “[email protected]”.

An OpenAI spokesperson told CyberScoop that the company is aware of the incident and is working with RubyGems and the researchers to conduct a broader review. The episode was characterized as “benign”.

“Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information,” the spokesperson said. “We’ll continue to investigate as part of our broader review of agent activity during training and evaluation.”

OpenAI said it has so far been unable to verify the researchers' specific claims about malicious packages or exploitation.

The incident came about two months before an autonomous AI agent breached part of Hugging Face's production infrastructure and accessed internal datasets and service credentials.