AI was once a developer’s best friend, but now it's a threat to the software supply chain

Approximately 97% of developers use generative artificial intelligence (AI) to write code, which causes a whole world of problems for developers and a new realm of opportunity for threat actors.

A recent study by researchers at the University of Texas at San Antonio, the University of Oklahoma, and Virginia Tech found that developers should be wary when using AI to code.

As developers rely mainly on programming languages such as Python and JavaScript, alongside centralized package repositories like Python Package Index (PyPI) and now code-generating large language models (LLMs), we’re seeing new issues emerge.

This has created what researchers call “a new type of threat to the software supply chain: package hallucinations.”

Package hallucinations happen when an LLM creates code that urges developers or refers developers to a package that doesn’t exist.

“These hallucinations, which arise from fact-conflicting errors when generating code using LLMs, represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain,” the study suggests.

The study looked at 16 major LLMs used for code generation and two prompt datasets to generate 576,000 code samples in two programming languages, which were analyzed to further understand package hallucinations.

Researchers found that the average percentage of hallucinated packages was at least 5.2% for commercial models like GPT-4 and Claude, and a whopping 21.7% for open-source models like CodeLlama and DeepSeek Coder.

They also noted that these included a “staggering 205,474 unique examples of hallucinated package names,” which further underscores the “severity and pervasiveness of this threat.”

While research on LLM hallucination and its implications for code generation is in its early stages, some studies have shown that popular LLMs like ChatGPT, CodeRL, and CodeGen “significantly hallucinate during code generation.”

This is particularly troubling, as threat actors could exploit this by creating a package with the same name as the hallucinated package and uploading it to an open-source repository like GitHub.

Users who are referred to this package might then use it in their code, thinking that it’s legitimate, while threat actors have injected the package with malicious code.

Those who are trusting of LLMs and are recommended the same hallucinated package over and over might end up downloading it, resulting in a compromise.

While this might not be the end of the world when it comes to independent projects, this compromise “can then spread through the entire codebase or software dependency chain, infecting any code that relies on the malicious package,” researchers warn.

This is what is known as a “package hallucination attack” and is a new and novel attack vector that threatens to “exacerbate the problem by exposing an additional threat surface for such attacks.”