The XZ Backdoor explained

As software projects become more complex, development teams are working to integrate different components from various sources to build their software solutions.

Leveraging software components from multiple vendors has many advantages, such as reducing development time and saving costs; however, it also comes with a risk: security.

This became clear after a recent cyberattack that exploited the software supply chain to plant a backdoor in a popular compression utility used by major Linux distributions.

At the end of March 2024, a Microsoft software developer known as Andres Freund sent a warning message to Openwall, a popular online forum dedicated to open-source software security.

The warning was about the discovery of a backdoor planted in a popular open-source compression utility called XZ utils package. XZ is an open-source lossless compression utility that comes preinstalled with major Linux distributions (e.g, Debian, Ubuntu, Fedora, and Arch Linux), including other UNIX-based systems and provides command line tools to work with the XZ compression format, including xz, unxz, xzcat, and xzgrep, in addition to supporting the legacy lzma format.

The XZ compression is popular among Linux/UNIX users because of its high compression ratio and excellent performance. The compression utility is used in different contexts in Linux/UNIX-based systems to compress files, archives, and software packages.

The discovery of this backdoor happened by coincidence. While Andres Freund was troubleshooting performance issues on a Debian operating system, he noticed the Debian system consumed a large volume of CPU cycles and generated errors with the Valgrind tool (a sophisticated tool for finding low-level programming errors in memory).

Andreas was accessing the compromised Debian system via SSH, a popular protocol for accessing systems remotely. After careful inspection of the reasons for consuming large CPU cycles, he discovered that the reason was a recent update to the XZ Utils that resulted in installing a backdoor in the system.

Technical specifications of the attack

Threat actors added malicious code (or a backdoor) to the XZ utility versions 5.6.0 and 5.6.1. The installed backdoor works by manipulating the sshd, a server process that facilitates secure internet connections using the SSH protocol. The sshd is responsible for user authentication, encryption, terminal connections, file transfers, and tunneling.

Now that the XZ backdoor has control over sshd, attackers can possess the encryption key used to make the SSH connections hide their malicious code in the SSH login certificate and use it as an entry point to execute various malicious actions on the compromised device – such as stealing files or installing malware (e.g., ransomware, keyloggers)

You might wonder how a program developed for compressing files could mess with the SSH protocol, which is very important in Linux-based systems. Well, let’s learn how it happens.

Think of your Linux computer system as a big city. SSH plays the role of a secure tunnel that lets people travel safely between different parts of the city. Now, the XZ utility is like a small company that provides a service in one part of the city – for example, a delivery company.

The "delivery company" is not directly related to the security of the tunnels. Still, it is connected to a bigger system that is responsible for managing services across different parts of the city, including the tunnels (SSH). This bigger system under Linux naming is called systemd. Systemd is a software suite (service manager and Linux initialization system) that provides many functions upon Linux booting.

Returning to our city example, when the XZ utility hooks into the systemd program, it's like a small service provider that suddenly gains access to the tunnels, which means the XZ utility now has access to the sshd process. This allows threat actors to remotely control affected systems by installing malicious programs or exfiltrating files without users' knowledge.

Who is behind the backdoor?

There seems to be a systematic approach behind this backdoor. In 2021, the threat actor behind the XZ backdoor opened a GitHub account named "JiaT75" (see Figure 1).

JiaT75's activities were not limited to their contributions to the XZ utility. For instance, the first contribution of this threat actor was in 2021 to the libarchive project.

Someone known as Lasse Collin is the original owner and developer of the XZ project. However, he has not contributed much to this repository in the last two years. In 2022, the malicious actor JiaT75 submitted a patch to the project (via its mailing list) that was irrelevant. Soon after, two other accounts named Jigar Kumar and Dennis Ens began to pressure "Collin" to add another maintainer to the ZX project (see Figure 2).

After three days of sending the pressure emails to "Collin," JiaT75 contributed their first commit to the XZ project. JiaT75 used a Gmail account: [email protected]

On January 7th, 2023, JiaT75 successfully merged its first commit into the XZ project. After two months, the primary contact email address in the OSS-FUZZ was updated to [email protected] (The threat actor's email address) (see Figure 3).

On February 10th, 2024, threat actor JiaT75 successfully changed the project domain name from tukaani.org/xz/ to xz.tukaani.org/xz-utils. The new website is now offline. However, by using the Wayback Machine, we can see previous captures of this website while the website is still alive (see Figure 4).

On February 23rd, the threat actor added a few files containing the remaining code needed to execute the backdoor (see Figure 5).

On March 29th, 2024, Software engineer Andres Freund discovered the issue and sent an alert to Openwall. The alert email subject was: backdoor in upstream xz/liblzma leading to ssh server compromise.

A Gist was also published on GitHub, containing highly technical details about this issue and what users should do to protect their compromised systems.