SolarWinds: The Poisoned Update That Reached Eighteen Thousand Networks

If NotPetya was a bomb going off in the update channel, SolarWinds was a listening device slipped into the same channel and left there, silent, for over a year. Both came through the software update everyone trusts. One was built to destroy as loudly as possible. The other was built to watch, and to make sure nobody ever knew it was watching. The second is the harder problem, and the more frightening one.

What was SolarWinds, and what happened?

SolarWinds is a company that makes IT management software. Its main product, called Orion, watches over an organization’s network: the servers, the devices, the traffic, the health of the whole system. To do that job, Orion needs deep access and high privileges, and it sits at the center of the network with a view of everything. That is exactly what made it such a perfect thing to poison. Software that watches everything is software that can be turned to watch everything for someone else.

Orion was used by a staggering roster of organizations: U.S. federal agencies including the Treasury, Homeland Security, the State Department, and parts of the defense and intelligence establishment, along with a large majority of the biggest publicly traded American companies. When you buy the tool that monitors your network, you install it deep and you trust it completely. Thousands of the most sensitive organizations in the country had done exactly that.

Beginning around 2019, a Russian intelligence hacking group, later attributed by the U.S. and U.K. governments to Russia’s foreign intelligence service and tracked under names like APT29 and Cozy Bear, broke into SolarWinds and did something more sophisticated than stealing data. They compromised the process by which Orion was built.

How do you poison a software update?

This is the technical heart of it, and it’s worth understanding because it defeats the defense most people assume protects them.

When a software company builds a new version of its product, the finished code goes through a build system that assembles it, and then it gets digitally signed. That signature is a seal of authenticity. It tells every customer’s computer that this update genuinely came from the vendor and wasn’t tampered with in transit. Your machine checks the signature, sees that it’s valid, and installs the update without complaint. The whole system of software trust rests on that signature.

The SolarWinds attackers didn’t forge the signature or tamper with the update after it left the building. They got inside the build system itself and inserted their malicious code before the software was compiled and signed.

So SolarWinds’ own process took the poisoned code, built it into Orion, and signed it with the company’s real, legitimate signature. The update that went out to 18,000 customers was authentic by every check a customer could run. It really did come from SolarWinds. It just also contained a back door, sealed inside the product with the vendor’s genuine seal of trust.

If that sounds familiar, it should. This is the exact lesson from the Stuxnet air gap, where stolen certificates made malicious drivers look genuine, and from Crypto AG, where the compromise was built into the product at the source. A valid signature answers the question “who made this,” and it does not answer the question “is this safe.” SolarWinds is that lesson written at the scale of the entire U.S. government.

Why did the SolarWinds back door take so long to catch?

Because it was built to be quiet, and because the people watching for intruders were looking in the wrong direction.

The back door, later named SUNBURST, was patient and careful. It lay dormant for a period after installation before doing anything. It disguised its own network traffic to look like normal, legitimate Orion communication, so that anyone monitoring the network saw nothing unusual. And, the sharpest touch, the attackers didn’t use it everywhere. Of the 18,000 organizations that installed the poisoned update, the hackers only activated the back door and moved deeper in a much smaller number of targets that genuinely interested them. A weapon used sparingly is a weapon that stays hidden.

The result is chilling. The intrusion went undetected for roughly 14 months. Federal agencies with serious security budgets, defense contractors, major corporations, none of them noticed the intruder living inside the tool they used to watch for intruders. The thing meant to be their eyes had been turned into the enemy’s eyes, and there is a bitter symmetry in that which I can’t read past quickly.

Who finally caught the SolarWinds intrusion?

Not by a government agency. Not by any of the thousands of Orion customers. By a private security company that realized it had been robbed.

The security firm FireEye detected an intrusion in its own network and, investigating, found that attackers had stolen some of its internal tools. Pulling that thread, its investigators traced the intrusion back to the poisoned SolarWinds update, and in December 2020 they went public. That disclosure set off a scramble across the government and the private sector, as thousands of organizations suddenly had to ask whether they were among the smaller set the attackers had chosen to exploit.

Think about what that means. The most consequential espionage breach of the U.S. government in years was not caught by the government’s own defenses. It was caught because the attackers made the mistake of also robbing a security company sharp enough to notice, and disciplined enough to follow the trail into its own embarrassment and then tell the world.

If FireEye had missed it, or stayed quiet to protect its reputation, the back door might still be open. Detection came down to one alert company and its willingness to disclose. That is far too thin a thread to hang national security on.

What does the SolarWinds attack teach that NotPetya didn’t?

NotPetya proved a poisoned update could destroy. SolarWinds proved a poisoned update could spy, patiently, at the highest level, and go unnoticed for more than a year. The two together define the shape of the trusted-update threat, and SolarWinds adds three lessons of its own.

The build system is the crown jewel. Everyone guards the finished product and the signing key, but the attackers went upstream of both, to the place where the product is assembled. If you can poison the code before it is built and signed, every downstream protection works in your favor, dutifully wrapping your attack in authenticity. Any organization that ships software has to treat its build pipeline as one of the most sensitive things it owns, and most did not before SolarWinds.

Detection has to assume prevention already failed. The victims weren’t careless. They ran a signed update from a trusted vendor. That is precisely correct behavior.

Prevention did everything it was supposed to do and still let the attacker in, because the attack was valid by every preventive check. What was missing was the assumption that something might already be inside, along with the network monitoring to find it. The only way anyone learns of an attack like this is by watching for the intruder’s behavior after the fact, because there is no bad click or failed signature to catch at the door.

And the whole thing turns on trust in a single vendor. Eighteen thousand organizations, including the government agencies charged with national security, handed deep access to their networks to one company’s software, so that company’s security became theirs. When it fell, they fell.

This is the pattern of every article in this series, from the Hezbollah pagers to Crypto AG to NotPetya, arriving now at the doorstep of the U.S. Treasury. The thing you trust to protect you becomes the thing that lets the attacker in, and the more completely you trust it, the more completely you’re exposed when it turns.

The next article brings the pattern down from nation-states to the everyday, through a retailer breached by way of its heating and cooling contractor. For the wider view of security and the people who need to understand it, the cybersecurity hub collects the rest of my work on this.

Frequently Asked Questions