SolarWinds: The Poisoned Update That Reached Eighteen Thousand Networks

If NotPetya was a bomb going off in the update channel, SolarWinds was a listening device slipped into the same channel and left there, silent, for over a year. Both came through the software update everyone trusts. One was built to destroy as loudly as possible. The other was built to watch, and to make sure nobody ever knew it was watching. The second is the harder problem, and the more frightening one.

SolarWinds is the attack in this series that stings most.

Every customer did what I’d tell any IT shop to do: they bought a reputable tool from an established vendor, kept it patched, and trusted the vendor’s signature. Doing the job right is exactly how the attackers got in. It’s a cruel twist, and I think anybody who has ever kept a network running will understand why.

What was SolarWinds, and what happened?

SolarWinds is a company that makes IT management software.

Its main product, called Orion, watches over an organization’s network: the servers, the devices, the traffic, the health of the whole system. To do that job, Orion needs deep access and high privileges, and it sits at the center of the network with a view of everything. That’s exactly what made it such a perfect thing to poison. Software that watches everything is software that can be turned to watch everything for someone else.

A staggering roster of organizations used Orion: U.S. federal agencies including the Treasury, Homeland Security, the State Department, and parts of the defense and intelligence establishment, along with a large majority of the biggest publicly traded American companies. When you buy the tool that monitors your network, you install it deep and you trust it completely. Thousands of the most sensitive organizations in the country had done exactly that.

I’d have bought Orion too, and that’s the uncomfortable part.

When a product sits inside the Treasury, Homeland Security and most of the biggest companies in the country, nobody in a budget meeting asks how well the vendor guards its build servers. They ask whether it works and what it costs. The question that mattered never came up, and I don’t think most buyers knew it was a question.

Beginning around 2019, a Russian intelligence hacking group, later attributed by the U.S. and U.K. governments to Russia’s foreign intelligence service and tracked under names like APT29 and Cozy Bear, broke into SolarWinds and did something more sophisticated than stealing data. They compromised the process by which Orion was built.

How do you poison a software update?

Learn this part if you’re responsible for anybody’s systems, because it defeats the check most IT departments treat as the final word. Once a signature validates, people stop asking questions, and the SolarWinds attackers were counting on that.

When a software company builds a new version of its product, the finished code goes through a build system that assembles it, and then it gets digitally signed.

That signature is a seal of authenticity. It tells every customer’s computer that this update came from the vendor and wasn’t tampered with in transit. Your machine checks the signature, sees that it’s valid, and installs the update without complaint. The whole system of software trust rests on that signature.

The SolarWinds attackers didn’t forge the signature or tamper with the update after it left the building. They got inside the build system itself and inserted their malicious code before the software was compiled and signed.

So SolarWinds’ own process took the poisoned code, built it into Orion, and signed it with the company’s real, legitimate signature.

The update that went out to 18,000 customers was authentic by every check a customer could run. It really did come from SolarWinds. It just also contained a back door, sealed inside the product with the vendor’s real seal of trust.

Think about the administrator who approved that update somewhere in a federal agency.

The checks passed and the vendor was trusted, so they installed it, and months later they learned they’d put a Russian intelligence back door on their own network with their own hands. Nothing on their screen could have warned them. I have a lot of sympathy for the people in that seat, who did their jobs correctly and still ended up as the ones who let the intruder in.

If that sounds familiar, it should. This is the exact lesson from the Stuxnet air gap, where stolen certificates made malicious drivers look genuine, and from Crypto AG, where the compromise was built into the product at the source. A valid signature answers the question “who made this,” and it doesn’t answer the question “is this safe.” SolarWinds is that lesson written at the scale of the entire U.S. government.

Why did the SolarWinds back door take so long to catch?

Because it was built to be quiet, and because the people watching for intruders were looking in the wrong direction.

The back door, later named SUNBURST, was patient and careful.

It lay dormant for a period after installation before doing anything. It disguised its own network traffic to look like normal, legitimate Orion communication, so that anyone monitoring the network saw nothing unusual. And, the sharpest touch, the attackers didn’t use it everywhere. Of the 18,000 organizations that installed the poisoned update, the hackers only activated the back door and moved deeper in a much smaller number of targets that interested them. A weapon used sparingly is a weapon that stays hidden.

The result is chilling. The intrusion went undetected for roughly 14 months. Federal agencies with serious security budgets, defense contractors, major corporations, none of them noticed the intruder living inside the tool they used to watch for intruders. The thing meant to be their eyes had been turned into the enemy’s eyes, and there’s a bitter symmetry in that which I can’t read past quickly.

Who finally caught the SolarWinds intrusion?

The answer should embarrass everyone who was supposed to be watching. No government agency caught it, and neither did any of the thousands of Orion customers. A private security company caught it because it realized it had been robbed.

The security firm FireEye detected an intrusion in its own network and, investigating, found that attackers had stolen some of its internal tools. Pulling that thread, its investigators traced the intrusion back to the poisoned SolarWinds update, and in December 2020 they went public. That disclosure set off a scramble across the government and the private sector, as thousands of organizations suddenly had to ask whether they were among the smaller set the attackers had chosen to exploit.

Think about what that means. The most consequential espionage breach of the U.S. government in years was caught only because the attackers made the mistake of also robbing a security company sharp enough to notice, and disciplined enough to follow the trail into its own embarrassment and then tell the world. If FireEye had missed it, or stayed quiet to protect its reputation, the back door might still be open. Detection came down to one alert company and its willingness to disclose. That’s far too thin a thread to hang national security on.

What does the SolarWinds attack teach that NotPetya didn’t?

NotPetya proved a poisoned update could destroy. SolarWinds proved a poisoned update could spy, patiently, at the highest level, and go unnoticed for more than a year. The two together define the shape of the trusted-update threat, and SolarWinds adds three lessons of its own.

The build system is the crown jewel. Everyone guards the finished product and the signing key, but the attackers went upstream of both, to the place where the product is assembled. If you can poison the code before it’s built and signed, every downstream protection works in your favor, dutifully wrapping your attack in authenticity. Any organization that ships software has to treat its build pipeline as among the most sensitive things it owns, and most didn’t before SolarWinds.

Detection has to assume prevention already failed. The victims ran a signed update from a trusted vendor, and no check at the front door was going to stop it. The only place left to catch it was inside the network, watching for traffic that didn’t belong.

Prevention did everything it was supposed to do and still let the attacker in, because the attack was valid by every preventive check. What was missing was the assumption that something might already be inside, along with the network monitoring to find it. The only way anyone learns of an attack like this is by watching for the intruder’s behavior after the fact, because there’s no bad click or failed signature to catch at the door.

And all of it turns on trust in a single vendor. Eighteen thousand organizations, including the government agencies charged with national security, handed deep access to their networks to one company’s software, so that company’s security became theirs. When it fell, they fell.

This is the pattern of every article in this series, from the Hezbollah pagers to Crypto AG to NotPetya, arriving now at the doorstep of the U.S. Treasury. The thing you trust to protect you becomes the thing that lets the attacker in, and the more completely you trust it, the more completely you’re exposed when it turns.

My verdict on SolarWinds is that most of the industry took the wrong lesson from it. The lazy response is to buy one more monitoring product and call the problem solved. The companies that learned something started asking vendors hard questions about how their software gets built, and started watching their own networks on the assumption that someone was already inside. A company that still treats a valid signature as proof of safety has ignored eighteen thousand warnings, and I’d tell its executives that to their faces.

The next article brings the pattern down from nation-states to the everyday, through a retailer breached by way of its heating and cooling contractor. For the wider view of security and the people who need to understand it, the cybersecurity hub collects the rest of my work on this.

Frequently Asked Questions