Malicious AI models infiltrating Hugging Face via ‘bad Pickles’

Hackers have found ways to hide malicious code in AI models hosted on the Hugging Face platform, ReversingLabs researchers warn.

At least two machine-learning models containing malicious code have been discovered on Hugging Face (HF). They bypassed HF’s security scanning mechanisms and weren’t flagged as unsafe. HF is a popular platform for hosting and accessing thousands of machine-learning models.

Hackers found a vulnerability in the way machine learning models are stored – the so-called Pickle files. Pickle files are used to store the model weights. However, threat actors can include malicious code that would run when developers try to run (deserialize) the model using Python.

That means hackers can insert any arbitrary malicious code, which could completely compromise the system and lead to data theft.

“Pickle is a popular Python module that many teams use for serializing and deserializing machine learning model data. While easy to use, Pickle is considered an unsafe data format, as it allows Python code to be executed during machine learning model deserialization,” the researchers explain in a report.

They call the novel attack technique “nullifAI,” as it evades existing protections.

The two discovered packages did not attempt to mimic popular legitimate AI models and appear to be proof-of-concept models. HF quickly disabled them after responsible disclosure. However, the risk for developers persists.

The HF security tool, Picklescan, failed to detect the malicious packages likely due to their compression format. By default, PyTorch (machine learning library for Python) uses the ZIP format for compression, and the malicious models were compressed using the 7z format, which can’t be loaded using the default torch.load() function.

Another discovered security flaw is the inability to properly scan broken Pickle files, which are still capable of executing the malicious payload.

“Threats lurking in Pickle files are not new. In fact, there are warnings popping out all over the documentation and there has been a lot of research on this topic,” the researchers noted.

They warn that attackers have many ways to exploit Pickle files and bypass the implemented security measures that are just waiting to be found.