In the realm of software security, the discovery of vulnerabilities is not uncommon. The Linux community was confronted with a particularly alarming revelation: a sophisticated backdoor had been unearthed within xz-utils, a fundamental suite of software utilized for lossless compression. This incident sent shockwaves throughout the community, underscoring the importance of robust security measures in safeguarding digital infrastructure. This blog endeavors to delve deeply into the intricacies of the backdoor, shedding light on its design, operation, and the ongoing efforts to mitigate its impact.

Analysis of the Backdoor Mechanism

The discovery of a backdoor in xz Utils versions 5.6.0 and 5.6.1 has raised significant concerns within the cybersecurity community. This section aims to dissect the functionality and development timeline of the backdoor, shedding light on its sophisticated design and potential implications.

Manipulation of SSHD

The backdoor ingeniously manipulates sshd, the executable responsible for facilitating remote SSH connections. By exploiting a predetermined encryption key, an attacker gains the ability to embed arbitrary code within an SSH login certificate. Subsequently, upon uploading and executing the certificate on the compromised device, the injected code executes, potentially enabling a myriad of malicious activities. While the exact nature of the code remains undisclosed, theoretical possibilities include theft of encryption keys, installation of malware, or unauthorized access to sensitive data.

Library Dependency Exploitation

The backdoor's implementation underscores the inherent risks associated with library dependencies within software ecosystems. Although OpenSSH, the prominent sshd implementation, does not directly link to the liblzma library, the situation becomes more intricate within certain Linux distributions. Debian and numerous others augment their implementations by linking sshd to systemd, a critical component responsible for orchestrating system services during bootup. As systemd relies on liblzma, a library provided by xz Utils, the backdoor gains an indirect pathway to exert control over sshd, exemplifying the nuanced interplay between software components within complex systems.

Development Timeline and Suspected Origins

The evolution of the backdoor appears to be a meticulously orchestrated endeavor, spanning several years and involving multiple actors. The timeline suggests a gradual infiltration of open-source projects, culminating in the implementation of the backdoor within xz Utils:

1. Initial Engagements (2021): The journey seemingly began with the appearance of a user named JiaT75, whose first known commit to an open-source project occurred in 2021. Subsequent alterations to the libarchive project, while inconspicuous at the time, hinted at a broader agenda.
2. Infiltration of XZ Utils (2022-2023): The following year saw JiaT75's active participation in discussions surrounding XZ Utils, advocating for increased development activity and the appointment of additional maintainers. With a gradual increase in involvement, JiaT75, later identified as Jia Tan, assumed a significant role within the project, leveraging their influence to implement the backdoor.
3. Implementation of the Backdoor (2023): In January 2023, Jia Tan initiated their first commit to XZ Utils, signaling the beginning of the backdoor's integration into the software. Over subsequent months, Tan's involvement intensified, culminating in the issuance of commits for versions 5.6.0 and 5.6.1, which incorporated the clandestine functionality.
4. Propagation and Outreach (2023-2024): Following the implementation of the backdoor, efforts were made to propagate the compromised versions across various Linux distributions. Appeals to developers of Ubuntu, Red Hat, and Debian sought the integration of the tainted updates into their respective operating systems, ultimately resulting in their inclusion in several releases.

Understanding the Backdoor

The backdoor embedded within xz-utils operates with a degree of subtlety that complicates detection. Initial investigations uncovered several key factors that contribute to system vulnerability:

1. Dependency on glibc: Systems relying on glibc, especially those leveraging the IFUNC mechanism, emerged as potential targets. This dependency provided a critical entry point for attackers seeking to exploit the system.
2. Specific xz-utils versions: Versions 5.6.0 and 5.6.1 of xz-utils or liblzma were identified as carriers of the backdoor, primarily affecting rolling-release distributions. This underscores the importance of diligent version management to mitigate risks.
3. Presence of systemd and patched openssh: Systems configured with systemd alongside patched openssh installations were deemed susceptible. However, further analysis is warranted to identify additional vulnerable configurations, highlighting the nuanced nature of the threat landscape.

The Design of the Backdoor

A closer examination of the backdoor's design reveals a meticulously orchestrated strategy aimed at circumventing security measures:

1. Altered release tarballs: Discrepancies between upstream release tarballs and their GitHub counterparts were observed, with modified versions of critical files injected into the former. This clandestine alteration obscured the true nature of the codebase, facilitating the surreptitious deployment of the backdoor.
2. Crafted test files: Malicious test files strategically embedded within the git repository served as triggers for the backdoor during the build process. These files remained dormant until invoked, demonstrating the covert nature of the attack vector.
3. Exploitation of IFUNC: The backdoor exploited IFUNC, a legitimate mechanism within glibc, to perform runtime hooking of OpenSSH's authentication routines. This insidious tactic enabled attackers to bypass authentication mechanisms, potentially granting unauthorized access to vulnerable systems.

Payload Analysis

While the precise intentions of the backdoor remain shrouded in uncertainty, preliminary analysis has yielded several noteworthy findings:

1. Activation via sshd: The backdoor activates when the running program matches the process name "/usr/sbin/sshd," suggesting a specific targeting of SSH services. This raises concerns about the potential for unauthorized access and exploitation of sensitive systems.
2. Remote code execution (RCE): Initial analysis indicates that the backdoor may facilitate remote code execution by allowing attackers to supply a key verified by the payload. This mechanism, if exploited, could lead to the execution of arbitrary commands, posing significant security risks.
3. Limited logging: Successful exploitation of the backdoor does not generate log entries, complicating detection and mitigation efforts. This stealthy behavior underscores the sophisticated nature of the attack vector and the challenges associated with identifying and neutralizing it.

Response and Mitigation:

In response to the security incident, concerted efforts are underway to address the backdoor and mitigate its impact:

1. Prompt updates: Linux distributions are issuing updates to patch vulnerabilities and safeguard against potential exploitation. Urgent action is being taken to mitigate the risk posed by the backdoor and protect vulnerable systems from compromise.
2. Enhanced scrutiny: Projects such as libarchive are undergoing thorough review to identify and rectify any associated vulnerabilities. This collaborative approach to security underscores the collective commitment to safeguarding the integrity of open-source software.
3. Tangential improvements: Discussions and proposals are emerging for enhancing security measures in related projects, such as CMake and systemd. These efforts aim to fortify defenses and minimize the risk of similar security incidents in the future.

Understanding and Responding to the Backdoored XZ Packages in Linux Distributions

The discovery of backdoored XZ packages in certain Linux distributions has raised concerns within the Linux community regarding potential security vulnerabilities. While not all distributions were affected, it is crucial for users to understand the implications and take necessary actions to mitigate risks.

Affected Distributions

1. Fedora: Fedora Rawhide and Fedora Linux 40 beta contained affected versions of the xz libraries (5.6.0, 5.6.1).
2. OpenSUSE: openSUSE Tumbleweed and openSUSE MicroOS included the affected xz version between March 7th and March 28th.
3. Debian: Debian testing, unstable, and experimental distributions had compromised xz-utils packages.
4. Kali Linux: Users who updated their installations between March 26th and March 29th are affected.
5. Arch Linux: Some Arch Linux virtual machine and container images, as well as an installation medium, contained the affected XZ versions.

Non-Affected Distributions

1. Red Hat: No versions of Red Hat Enterprise Linux (RHEL) are affected.
2. Ubuntu: No released versions of Ubuntu were affected.
3. Linux Mint, Gentoo Linux, Amazon Linux, and Alpine Linux were not affected by the backdoor.

Mitigation Steps

1. Follow Distribution Guidance: Users should adhere to the guidance provided by the maintainers of their Linux distribution. This may involve updating affected packages or taking other remedial actions recommended by distribution maintainers.
2. Check System: Users are advised to run scripts provided by distribution maintainers or security experts to check if their system utilizes backdoored versions of the liblzma library.
3. Treat as Security Incident: Any system with affected packages installed should be treated as a potential security incident. Users are encouraged to investigate thoroughly to determine if the backdoor was exploited.
4. Review and Rotate Credentials: It is recommended to review all sensitive information and credentials stored on the machine. Any compromised credentials should be rotated promptly to prevent unauthorized access.
5. Assess Blast Radius: Users should assess the potential impact of the security incident on other assets within the system's "blast radius." This involves identifying and securing any related assets that could be affected by the compromised machine.

Verifying Protection: XZ Utils Backdoor Security Check

Thanks to Vegard Nossum for Providing the Script in the Disclosure.

#! /bin/bash
 
set -eu
 
# find path to liblzma used by sshd
path="$(ldd $(which sshd) | grep liblzma | grep -o '/[^ ]*')"
 
# does it even exist?
if [ "$path" == "" ]
then
	echo probably not vulnerable
	exit
fi
 
# check for function signature
if hexdump -ve '1/1 "%.2x"' "$path" | grep -q f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
then
	echo probably vulnerable
else
	echo probably not vulnerable
fi

Ensure it's executable, then execute the script:

cyberunfolded:~$ chmod +x detect.sh
cyberunfolded:~$ ./detect.sh
probably not vulnerable

Resumption of Control

Lasse Collin, recognized as the longstanding steward of XZ Utils, has played a pivotal role in its development and maintenance since its inception. However, amidst the infiltration of the project by malicious actors and the implementation of the backdoor, Collin's control over the project was temporarily compromised. Despite his absence during the initial stages of the incident, Collin's expertise and dedication to the project remained undiminished.

Upon learning of the security breach and the presence of the backdoor, Collin swiftly moved to reclaim control of the project. His return to the helm signifies a resolute commitment to restoring trust, rectifying vulnerabilities, and fortifying the integrity of XZ Utils. Collin's extensive knowledge of the project's intricacies positions him as a linchpin in the ongoing efforts to purge the codebase of malicious elements and fortify its defenses against future threats.

Cleaning the Code and Collaborative Efforts

Collin's foremost priority upon reassuming control is the meticulous cleaning and fortification of the XZ Utils codebase. Drawing upon his deep understanding of the project's architecture and functionality, Collin undertakes a comprehensive review and audit of the code, identifying and eliminating any remnants of the backdoor and associated vulnerabilities. Through methodical scrutiny and rigorous testing, Collin aims to restore the project to its pristine state, free from malicious tampering and exploitation.

In tandem with his individual efforts, Collin collaborates closely with the broader open-source community, leveraging collective expertise and resources to expedite the cleanup process. By fostering open communication channels and soliciting feedback from fellow developers and contributors, Collin ensures a transparent and inclusive approach to code remediation. This collaborative ethos fosters a sense of collective responsibility and solidarity, reinforcing the community's resilience in the face of adversities.

Conclusion

The discovery of the backdoor in xz-utils serves as a sobering reminder of the persistent threat landscape facing software ecosystems. Through meticulous analysis, collaboration, and proactive measures, the Linux community strives to fortify defenses, mitigate risks, and uphold the integrity of open-source software. As investigations continue and remediation efforts unfold, maintaining transparency, resilience, and a shared commitment to security remain paramount. By remaining vigilant, informed, and proactive, we can navigate through challenges, strengthen our defenses, and preserve the trust and integrity of digital infrastructure.