The XZ Backdoor: Two Years of Patience, Caught by Luck

Everything the modern world runs on is built, in part, out of small pieces of free software that volunteers write and maintain.

Not the big famous programs. The plumbing. The little utilities that compress a file, handle a connection, parse a date, buried so deep in the machinery that almost nobody thinks about them, running on servers and phones and devices in numbers no one can count. One or two unpaid people maintain much of this critical plumbing in their spare time.

Starting in 2021, an attacker went hunting in exactly that blind spot. Of all the attacks in this series, this is the one I can’t shake, because nobody broke anything. The attacker was patient and helpful, and picked on the one person holding the project up, an unpaid volunteer who was tired. I find that uglier than any worm or poisoned update, and I think the companies that built fortunes on that volunteer’s work share the blame. (More on billions for AI, pennies for the plumbing)

What did the XZ backdoor let attackers do?

XZ Utils is one of those invisible pieces of plumbing. It’s a compression tool, software that squeezes files down to a smaller size, and it is so standard that it ends up installed, directly or through other programs, on a huge share of the computers running Linux, the operating system that powers most of the servers on the internet. You’ve never heard of it, and you almost certainly rely on it every day.

In early 2024, a hidden backdoor was discovered inside recent versions of XZ Utils. It was built so that an attacker holding a specific secret key could reach through the compression library into a common remote-login program and run their own commands on the machine, silently, bypassing the normal password checks. In plain terms: whoever planted it would have been able to break into an enormous number of computers around the world, at will, and nobody would have known how.

Security professionals rate vulnerabilities on a scale up to ten. This one was rated a ten, the maximum, and the rating almost undersells it. This was a weapon someone deliberately built into the software, wearing the disguise of a routine improvement, and it was days from spreading into the stable versions that the whole world installs.

Anyone who has run production servers knows what a back door into remote login means. You can’t trust a single machine you own, you can’t trust your logs, and you can’t know what was taken, so you rebuild from scratch and hope your backups are older than the infection. I wouldn’t wish that week on anyone, and this back door would have handed it to much of the internet at once.

How do you plant a backdoor in software the whole world uses?

The attacker got in the way a con artist gets into a lonely neighbor’s finances, by being helpful for a long time. That move sets XZ apart from everything else in this series. XZ Utils, like so much critical open-source software, depended largely on one person, a volunteer who had kept it running for years and, by his own accounts, was stretched thin and dealing with his own life.

In 2021, an account under the name “Jia Tan” appeared and began contributing to the project.

Not maliciously, at first. Helpfully. Fixing things, submitting improvements, doing the tedious work that an overstretched maintainer is grateful for. Over roughly two years, this contributor made hundreds of contributions and slowly, patiently, built a reputation as a trusted, valuable member of the project.

Then came the pressure. Alongside the helpful work, other accounts appeared in the project’s discussions, complaining that the software wasn’t being updated fast enough, that the lone maintainer wasn’t keeping up, that the project needed someone more active in charge.

These accounts, widely believed to be part of the same operation, sockpuppets working the same con, pushed on exactly the sore spot of a tired volunteer: you’re not doing enough, and here’s someone who is. The campaign worked. The maintainer, worn down and looking for help, gave Jia Tan expanded control over the project.

Those complaint threads make me angry every time I read them. Somebody deliberately ground down a volunteer doing a thankless job for free, shamed him in public for not doing more, and then offered the cure. The pressure campaign was the cruelest part of the whole operation, and it worked because nobody else was standing next to him.

With the keys in hand, the attacker moved carefully. The malicious code went into the packaged release files, the compiled bundles most systems install from, where anyone reviewing the project’s public source wouldn’t see it. It was tucked into build scripts and disguised test files in a way designed to survive a normal review.

The trap was set, built into official releases, signed and shipped as a legitimate update from a trusted project, and it began making its way toward the stable versions that every major Linux system would soon adopt.

How did one engineer stumble onto the XZ backdoor?

One person caught it by accident, by noticing that something was a little slow. A software engineer named Andres Freund, who worked at Microsoft, was doing unrelated performance testing when he noticed that logging into a machine over the network was taking about half a second longer than it should. Half a second. Most people would never notice, and of the few who did, almost none would chase it.

Freund chased it. He dug into why the login was slow, found that a compression library was eating an odd amount of processor time, kept pulling the thread, and uncovered the backdoor hidden inside XZ Utils. He reported it publicly at the end of March 2024, and the security world’s weekend erupted.

Think about how thin that margin was. One careful engineer caught the most sophisticated software supply chain attack anyone had seen, years in the making and aimed at much of the internet’s infrastructure, because a half-second delay bothered him and he had the curiosity and the skill to run it down.

Had he shrugged, or been busy that week, the backdoored versions would have flowed into the stable releases and onto machines worldwide, and we would be telling a very different story, if we knew to tell it at all. There was no system that caught this. There was a person, and a coincidence.

That should embarrass the whole security industry. Enormous sums go into security products every year, and a software engineer doing unrelated testing, who didn’t like a half-second delay, found the most dangerous back door in years. I’d take one person with that kind of curiosity over any product on the market.

Why is the XZ backdoor the most unsettling attack in this series?

Because it inverts the pattern of every other article here, and the inversion is worse.

In the pager attack, Israel built a company to become a trusted supplier.

In Crypto AG, intelligence agencies bought a company to control a trusted product. Those took money, infrastructure, a nation behind them. The XZ attack achieved the same thing, trusted-supplier access to a critical piece of the world’s software, with almost none of that. It took one persistent person, a couple of fake accounts, and two years of patience. The barrier to becoming trusted was time and social pressure applied to a tired volunteer.

That should change how you think about the software running your life, because the same structural weakness is everywhere. People who are unpaid, overworked, and largely unsupported maintain a staggering amount of the code that runs the modern world, holding up infrastructure that trillion-dollar companies and governments depend on. Every one of those maintainers is a target for exactly the campaign that hit XZ: befriend, help, pressure, take over. We got lucky once, publicly. The question is how many times we haven’t.

The companies that run on this code should be ashamed of how little they give back. They spend fortunes on their own products, leave a critical library to one worn-out volunteer, and act surprised when someone exploits the gap. If your business depends on open-source plumbing, find out who maintains it and help pay for that work. It costs less than one incident response, and it’s the decent thing to do.

Every other article in this series shows an attacker defeating the victim’s trust: crossing the air gap, forging the certificate, poisoning the update.

XZ is the version where the attacker doesn’t defeat trust at all.

They earn it, legitimately, through work indistinguishable from a good contributor’s, and then use it. There’s no technical control that catches that, because at every step the attacker looked exactly like what a trustworthy participant looks like. The only defense is human judgment about who is being trusted with what. That’s exactly the defense that doesn’t scale, and exactly the one the final article in this series argues we’re now handing to machines.

My verdict is that XZ is a people failure dressed up as a software story. We left critical infrastructure in the hands of exhausted volunteers, gave them almost no support, and a patient attacker walked through the gap we made. Until the companies profiting from open source carry some of that weight, I expect to read this story again, and I doubt a half-second delay will give it away next time.

The last article steps back and asks what all twelve of these attacks, taken together, say about the AI systems being built right now, and about the trust we’re placing in things whose behavior no one fully understands. For the wider view, the cybersecurity hub collects the rest of my work on security.

Frequently Asked Questions