A layman's perspective on the xz debacle

March 31, 2024

On Friday, March 29, 2024, the Linux community was hit with a devastating blow in the form of malware being discovered in a heavily relied upon piece of software, though the impact was mostly to its ego. The problem stemmed from malicious code being added to the xz package, a compression utility used by the vast majority of Linux distributions and which is incorporated in the kernel itself. The malware would cripple the security on some systems by compromising the sshd daemon.

The versions of xz known to be compromised are 5.6.0 and 5.6.1, however earlier versions which included commits from the "JiaT75" GitHub account, which was responsible for the malware, may also be partially compromised. The shell script, xz_cve-2024-3094-detect.sh, does some rudimentary testing to detect whether a system may be vulnerable.

Security Alert: Potential SSH Backdoor Via Liblzma | Hackaday

What may not be clear is the connection to SSH. And it's a trip. Many Linux distros patch sshd to add systemd features, and libsystemd pulls the liblzma library. That means the liblzma initialization code gets run when sshd starts. In the malicious code, the library checks argv[0], which is the name of the program being executed, for /usr/bin/sshd. Additionally it seems to check for debugging tools like rr and gdb. If the checks are green, liblzma replaces a few function calls with its own code. It's a complicated dance, but the exploit is specifically looking to replace RSA_public_decrypt.
That's a very interesting function to clobber, as it is one of the functions used to validate SSH keys.

CVE Record | CVE

Malicious code was discovered in the upstream tarballs of xz, starting with version 5.6.0. Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code. This results in a modified liblzma library that can be used by any software linked against this library, intercepting and modifying the data interaction with this library.

What is particularly devastating about this malware is not it's exposure, which was thankfully very limited to cutting edge releases such as Debian's and Fedora's unstable branches, but rather that a widely distributed and heavily used bit of software was severely compromised without anyone noticing. This has some folks wondering what other projects may have been successfully infiltrated by those using similar methods.

A general overview and timeline of this entire fiasco is available on the Everything I know about the XZ backdoor page while a more detailed one is available on the Timeline of the xz open source attack page. Following is an early info-graphic by Thomas Roccia which provides the vitals of the problem as they were known at the time. Thomas adds the following note:

I hope it helps to make sense of the information out there. Please treat the information "as is" while the analysis progresses!

The root of the problem, from my perspective, seems to be that the xz project was maintained by a sole developer, Lasse Collin, who tells us he's been experiencing difficulties in his life which apparently led to a degree of apathy regarding a crucial project he's been working on for decades, without pay, and which is utilized by virtually every Linux distribution.

Re: [xz-devel] XZ for Java

I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.
It's also good to keep in mind that this is an unpaid hobby project.

Collin accepted help in the form of a very sneaky and talented "Jia Tan" who wasn't properly vetted and who, over the course of roughly 3 years, worked on gaining trust and then implementing the back-door after being accepted as a co-maintainer for the xz project. With the trust of Collin won, the first commit from the Jia Tan account was made in early 2022 while the final commits followed approximately 2 years later. Through systemd, the hack would compromise sshd if certain conditions are met, thus allowing remote code execution for any affected box running an ssh server.

What is known as of this writing ... is that there's quite a bit that isn't yet known. Due to the complexity of the code, reverse engineering the binary is proving to be a tough nut to crack. The full extent of the developers intentions and the code itself don't seem to be completely understood, though the target seems to have been Debian and Red Hat systems. Developer Sam James writes the following:

This attack thusly seems to be targeted at amd64 systems running glibc using either Debian or Red Hat derived distributions. Other systems may be vulnerable at this time, but we don't know.
[...]
We don't know what the payload is intended to do. We are investigating.

Though the malware was picked up by various rolling release distributions, including Arch, the consensus seems to be that these systems are probably not at risk. The average desktop user, regardless of which distro they use, should be unaffected since they will not typically be running an ssh server, but again, there's much that isn't yet fully understood.

All of the major Linux distributions have issued security warnings for CVE-2024-3094 and pushed a different version of xz to their repositories where necessary (again, mostly the rollers), so it would certainly be prudent to update your system, like immediately.

Urgent security alert for Fedora 41 and Fedora Rawhide users

What is the malicious code?
The malicious injection present in the xz versions 5.6.0 and 5.6.1 libraries is obfuscated and only included in full in the download package - the Git distribution lacks the M4 macro that triggers the build of the malicious code. The second-stage artifacts are present in the Git repository for the injection during the build time, in case the malicious M4 macro is present.
The resulting malicious build interferes with authentication in sshd via systemd. SSH is a commonly used protocol for connecting remotely to systems, and sshd is the service that allows access. Under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.
What distributions are affected by this malicious code?
Current investigation indicates that the packages are only present in Fedora 40 and Fedora Rawhide within the Red Hat community ecosystem.
No versions of Red Hat Enterprise Linux (RHEL) are affected.
We have reports and evidence of the injections successfully building in xz 5.6.x versions built for Debian unstable (Sid). Other distributions may also be affected. Users of other distributions should consult with their distributors for guidance.

Arch Linux - News: The xz package has been backdoored

TL;DR: Upgrade your systems and container images now!
As many of you may have already read (one), the upstream release tarballs for xz in version 5.6.0 and 5.6.1 contain malicious code which adds a backdoor.
[...]
Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:

ldd "$(command -v sshd)"

However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.

The malware was discovered by Andres Freund who is a developer for 'the enemy' (Microsoft). Nonetheless, hats off to Mr. Freund! xoxo

After observing a few odd symptoms around liblzma (part of the xz package) on
Debian sid installations over the last weeks (logins with ssh taking a lot of
CPU, valgrind errors) I figured out the answer:
The upstream xz repository and the xz tarballs have been backdoored.
At first I thought this was a compromise of debian's package, but it turns out
to be upstream.

The community consensus seems to be that Lasse Collin was not actively involved in the development of the malware and that it was instead his assigned co-maintainer who used the nick Jia Tan (JiaT75 on github, jiat0218@gmail.com). Collin was criticized for not updating the project and pressured to add another maintainer and incorporate changes to the code at a vulnerable time in his life by several very suspicious accounts which seem to have been created specifically for this purpose, possibly all by the same person who could also be masquerading as Jia Tan. The following comments are from one of these highly suspicious accounts belonging to Jigar Kumar:

The only progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?

Progress will not happen until there is new maintainer. XZ for C has sparse commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this.

Given that Jia Tan had been working on xz for roughly 2 years, and given the complexity of the hack, many suspect that "Jia Tan" is actually a state actor and not a single person, or perhaps a person or entity who planned to monetize the exploit which could have been worth a massive heap of shekels.

The level of commitment, patience and sophistication demonstrated by the attacker is stunning. The malware is distributed among multiple files in multiple locations, and even multiple parts of a single file, and is then assembled via a wildly obfuscated shell script. It appears the malware is quite selective and dependent upon, among other things, systemd, though only indirectly so. Nevertheless this has several people bashing systemd, again, and though i'm personally not a fan of the systemd virus (and pretty much hate Lennart Poettering, who is defending the monstrosity he created), i'm not sure those developing systemd should be implicated. Also the question needs to be asked: if systemd didn't exist, would Jia Tan not have found another way to incorporate the back-door? The following is another comment from developer Sam James:

The payload is loaded into sshd indirectly. sshd is often patched to support systemd-notify so that other services can start when sshd is running. liblzma is loaded because it's depended on by other parts of libsystemd. This is not the fault of systemd, this is more unfortunate.

A breakdown of how this complex bit of malware is assembled can be found at xz/liblzma: Bash-stage Obfuscation Explained - gynvael.coldwind//vx.log and Midar/xz-backdoor-documentation Wiki.

Another very interesting aspect of this hack is that there appears to be a killswitch implemented in the code.

There appears to be a string encoded in the binary payload:
https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01#file-hashes-txt-L115
Which functions as a killswitch:
https://piaille.fr/@zeno/112185928685603910
Thus, one workaround for affected systems might be to add this to `/etc/environment`:
yolAbejyiejuvnup=Evjtgvsh5okmkAvj
+ restart ssh and systemd

There is no shortage of speculation as to who, or what, "Jia Tan" is with China, Russia and Israel often being implicated as the countries of origin, however Rhea Karty and Simon Henniger have done a very interesting and detailed analysis of the commit time stamps to the xz project by Jia Tan, who wants us to believe "he" is Chinese. The XZ Backdoor: Times, damned times, and scams article on Substack potentially places Jia Tan in Eastern Europe in the UTC +2/3 time zone where i think the countries of primary interest are Russia and Israel. Interestingly, Jia Tan, Hans Jansen and Lasse Collin all seem to reside in the same time zone according to the time stamp analysis.

Chinese bank holidays (just looking at 2023):
- Working on 2023, 29 September: Mid Autumn Festival
- Working on 2023, 05 April: Tomb Sweeping Day
- Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year
Eastern European holidays:
- Never working on Dec 25: Christmas (for many EET countries)
- Never working Dec 31 or Jan 1: New Years
To further investigate, we can try to see if he worked on weekends or weekdays: was this a hobbyist or was he paid to do this? The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79).
Again, none of this is rock-solid evidence for anything - but we thought this was an interesting enough observation to post. Based on this, "Jia" most likely worked regular office hours and was based somewhere in UTC+02/03 (e.g. EET).

The fallout from the xz disaster has developers moving to eliminate their reliance upon xz, including those who work on the Linux firmware updater.

Linux Firmware Update Utility Fwupd Will Use Zstd Compression for Future Releases - 9to5Linux

Fwupd developer and maintainer Richard Hughes announced today that future releases of the popular Linux firmware updater used by GNU/Linux distributions to update the firmware of various hardware devices are moving away from XZ Utils and will adopt Zstandard (zstd) instead.
After the XZ backdoor fiasco, now open-source developers are looking for an alternative compression utility, and the obvious choice these days is Zstandard (also known as zstd for short), which provides a lossless data compression algorithm that proves to be faster than XZ when decompressing.
Zstandard was developed by Yann Collet at Facebook. Apart from being known for fast compression, it also provides high compression ratios compared to XZ. According to Richard Hughes, the fwupd metadata compressed with zstd is around 3 percent smaller than the one compressed with XZ.
However, the real benefit of using Zstandard for compressing fwupd metadata is that the developers trust it a lot more than XZ now, and this is just the beginning because more and more open-source projects will start adopting zstd to ensure the safety of their users.

Update, 1-May-2024

The following lecture by Denzel Farmer is a deep-dive into the xz debacle presented to the W4995 Advanced Systems Programming class at Columbia University. The lecture appears to have been given around April 24, 2024. The PDF is available on archive.org.

Deep Dive into XZ Utils Backdoor - Columbia Engineering, Advanced Systems Programming Guest Lecture

On March 29th, a developer from Microsoft published that he had discovered a backdoor built into XZ Utils, a compression package included with nearly every major Linux distribution. If gone unnoticed, this backdoor could have provided its authors with root-level access to millions servers across the internet. Interestingly, the core mechanism the backdoor uses to compromise host machines is something we just finished studying - dynamic linking and loading of ELF objects. This lecture will explore implementation details of the XZ Utils backdoor and describe the novel multi-year effort to put it in place-along with its consequences for the larger world of open source software development.
This is a recording of a guest lecture presented to the W4995 Advanced Systems Programming class at Columbia University.
00:00 - Intro
02:09 - Background on Open Source Development
05:47 - Backdoor Timeline
19:31 - How the Payload Works
48:46 - Reverse Engineering the Payload
57:56 - Live Demo
1:01:35 - Attribution
1:05:37 - Larger Implications

Resources: