The Infection That Kept Coming Back

One infection at the retailer taught me more about incident response than any course ever did, because it refused to die. It was a network-propagating piece of malware, and our first response was the obvious one: find an infected machine, clean it, move to the next.

The malware had a different plan. Clean machine A. While you’re cleaning machine B, machine C reinfects machine A. Clean C, and by then the infection has looped back through D and E and is sitting on A and B again. We were playing whack-a-mole against an opponent that never got tired, never went home, and worked every machine at the same time while we worked one at a time.

That incident still looks sloppy to me, mostly because of how long it took us to stop doing the obvious thing. Cleaning machines one at a time feels like progress. Every technician is busy, every desk visit ends with a clean machine, and the effort looks impressive while the infection wins. I’ve come to distrust any incident response that looks productive before anyone has checked whether it’s working.

The arithmetic of losing

The failure was mathematical. Our cleaning rate was serial: one technician, one machine, some number of minutes. The malware’s infection rate was parallel: every infected machine was a transmitter, all the time.

When the propagation rate exceeds the cleaning rate, the infected population grows while you work. You can staff up, and we did, but you’re racing an exponential with a linear tool.

Every IT team that’s fought a network worm knows this feeling, and most learn the answer the same way we did: the hard way. I think that’s a failure in how the field trains people. Nobody should have to burn a week of users’ time rediscovering arithmetic that fits on a napkin.

We tried the obvious countermove first: more hands.

Pull technicians from other work, clean faster. It helped the way bailing helps a holed boat, meaning it changed the numbers without changing the outcome. Parallel infection beats serial cleaning at any staffing level you can afford, because the malware recruits every machine it captures while each technician remains exactly one technician. The staffing response fails mathematically before it fails operationally, and recognizing that early saves you the week we spent proving it.

Pulling technicians off their other work also exposed my biggest complaint about how companies organize this.

At the retailer, security was part of IT, and it caused problems because people put their own IT issues ahead of security. Executives lump the two together, and they shouldn’t. Security belongs in its own department, possibly reporting to the COO while IT usually reports to the CFO, so its judgment about a threat isn’t bent by an IT backlog. IT thinks about IT issues. Security has to think about people.

How do you stop a malware reinfection loop?

The cycle broke when we changed the unit of response. Instead of cleaning machines inside a hostile network, we isolated the workstation network itself, then split the population: a clean segment for disinfected machines, a dirty segment for everything else. Machines moved one direction only, dirty to clean, and only after verification. The transmitters could no longer reach the cured.

Once separation was in place, the numbers flipped. Every cleaned machine was permanently subtracted from the infected population instead of temporarily. The dirty segment shrank monotonically to zero, and the incident, which had run for a long, demoralizing stretch, ended in an orderly march.

I wish we’d done it on day one. Every hour we spent cleaning inside a hostile network was an hour users sat without machines while technicians did work the malware undid behind them. That waste belongs to whoever is running the response, and on that incident it belonged to us.

Why does malware keep coming back?

Containment precedes eradication. That ordering is now standard incident-response doctrine, but doctrine reads like a slide until you’ve watched a cleaned machine light up again ninety seconds after you left the desk. The lived version of the rule is simpler: never fight an infection on ground the infection controls. Take the ground first.

The same ordering governs modern incidents. Ransomware response starts with isolating segments, and restoring files comes after. Compromise response starts with cutting attacker access. Resetting passwords one account at a time while the attacker still holds the mailbox that receives the resets is serial cleanup all over again. Every generation of defenders rediscovers the ordering, usually mid-incident, and I find that maddening, because it’s been written down for decades.

When executives ask me what makes a security war story worth publishing, this is my example.

The malware wasn’t exotic. The failure mode was serial defense against a parallel attacker, and it’s universal, so a reader who absorbs it will recognize it in incidents that haven’t happened yet. I’d rather read one executive admitting to a week of whack-a-mole than a hundred books pretending their teams got everything right the first time, because the pretending teaches nobody anything.

For more from this series, see The Cybersecurity Hub: breaches, audits, and hard-won security lessons from 33 years in the trenches.

Frequently Asked Questions