The XZ Backdoor: Two Years of Patience, Caught by Luck

Everything the modern world runs on is built, in part, out of small pieces of free software written and maintained by volunteers. Not the big famous programs. The plumbing. The little utilities that compress a file, handle a connection, parse a date, buried so deep in the machinery that almost nobody thinks about them, running on servers and phones and devices in numbers no one can count. Much of this critical plumbing is maintained by one or two unpaid people in their spare time.

In 2024, an attacker went hunting in exactly that blind spot, and the story of what they did and how close they came should frighten anyone who depends on a computer. That means everyone.

What did the XZ backdoor let attackers do?

XZ Utils is one of those invisible pieces of plumbing. It’s a compression tool, software that squeezes files down to a smaller size, and it is so standard that it ends up installed, directly or through other programs, on a huge share of the computers running Linux, the operating system that powers most of the servers on the internet. You have never heard of it, and you almost certainly rely on it every day.

In early 2024, a hidden backdoor was discovered inside recent versions of XZ Utils. It was built so that an attacker holding a specific secret key could reach through the compression library into a common remote-login program and run their own commands on the machine, silently, bypassing the normal password checks. In plain terms: whoever planted it would have been able to break into an enormous number of computers around the world, at will, and nobody would have known how.

Security professionals rate vulnerabilities on a scale up to ten. This one was rated a ten, the maximum, and the rating almost undersells it. This wasn’t a flaw somebody found in existing software. It was a weapon someone deliberately built into the software, wearing the disguise of a routine improvement, and it was days from spreading into the stable versions that the whole world installs.

How do you plant a backdoor in software the whole world uses?

You don’t break in. You get hired, in effect, by being helpful. That move sets XZ apart from everything else in this series.

XZ Utils, like so much critical open-source software, was maintained largely by one person, a volunteer who had kept it running for years and, by his own accounts, was stretched thin and dealing with his own life.

In 2021, an account under the name “Jia Tan” appeared and began contributing to the project. Not maliciously, at first. Helpfully. Fixing things, submitting improvements, doing the tedious work that an overstretched maintainer is grateful for. Over roughly two years, this contributor made hundreds of contributions and slowly, patiently, built a reputation as a trusted, valuable member of the project.

Then came the pressure. Alongside the helpful work, other accounts appeared in the project’s discussions, complaining that the software wasn’t being updated fast enough, that the lone maintainer wasn’t keeping up, that the project needed someone more active in charge.

These accounts, widely believed to be part of the same operation, sockpuppets working the same con, pushed on exactly the sore spot of a tired volunteer: you’re not doing enough, and here is someone who is. The campaign worked. The maintainer, worn down and looking for help, gave Jia Tan expanded control over the project.

With the keys in hand, the attacker moved carefully. The malicious code was not dropped in plainly where anyone reviewing the project’s public source would see it. It was hidden in the packaged release files, the compiled bundles most systems install from, tucked into build scripts and disguised test files in a way designed to survive a normal review.

The trap was set, built into official releases, signed and shipped as a legitimate update from a trusted project, and it began making its way toward the stable versions that every major Linux system would soon adopt.

How did one engineer stumble onto the XZ backdoor?

By accident. By one person noticing that something was a little slow.

A software engineer named Andres Freund, who worked at Microsoft, was doing unrelated performance testing when he noticed that logging into a machine over the network was taking about half a second longer than it should. Half a second. Most people would never notice, and of the few who did, almost none would chase it.

Freund chased it. He dug into why the login was slow, found that a compression library was eating an odd amount of processor time, kept pulling the thread, and uncovered the backdoor hidden inside XZ Utils. He reported it publicly at the end of March 2024, and the security world’s weekend erupted.

Sit with how thin that margin was. The most sophisticated software supply chain attack anyone had seen, years in the making, aimed at much of the internet’s infrastructure, was caught because one careful engineer was bothered by a half-second delay and had the curiosity and the skill to run it down.

Had he shrugged, or been busy that week, the backdoored versions would have flowed into the stable releases and onto machines worldwide, and we would be telling a very different story, if we knew to tell it at all. There was no system that caught this. There was a person, and a coincidence.

Why is the XZ backdoor the most unsettling attack in this series?

Because it inverts the pattern of every other article here, and the inversion is worse.

In the pager attack, Israel built a company to become a trusted supplier. In Crypto AG, intelligence agencies bought a company to control a trusted product. Those took money, infrastructure, a nation behind them. The XZ attack achieved the same thing, trusted-supplier access to a critical piece of the world’s software, with almost none of that. It took one persistent person, a couple of fake accounts, and two years of patience. The barrier to becoming trusted was not a billion dollars. It was time and social pressure applied to a tired volunteer.

That should change how you think about the software running your life, because the same structural weakness is everywhere. A staggering amount of the code that runs the modern world is maintained by people who are unpaid, overworked, and largely unsupported, holding up infrastructure that trillion-dollar companies and governments depend on. Every one of those maintainers is a target for exactly the campaign that hit XZ: befriend, help, pressure, take over. We got lucky once, publicly. The honest question is how many times we haven’t.

The deepest link to the rest of this series, and to where it’s heading, is this. Throughout these articles, the attacker had to defeat the victim’s trust, cross the air gap, forge the certificate, poison the update.

XZ is the version where the attacker doesn’t defeat trust at all. They earn it, legitimately, through work indistinguishable from a good contributor’s, and then use it. There is no technical control that catches that, because at every step the attacker looked exactly like what a trustworthy participant looks like. The only defense is human judgment about who is being trusted with what. That is precisely the defense that does not scale, and precisely the one the final article in this series argues we are now handing to machines.

The last article steps back and asks what all twelve of these attacks, taken together, say about the AI systems being built right now, and about the trust we are placing in things whose behavior no one fully understands. For the wider view, the cybersecurity hub collects the rest of my work on security.

Frequently Asked Questions