The Infection That Kept Coming Back

One infection at the retailer taught me more about incident response than any course ever did, because it refused to die. It was a network-propagating piece of malware, and our first response was the obvious one: find an infected machine, clean it, move to the next.

The malware had a different plan. Clean machine A. While you are cleaning machine B, machine C reinfects machine A. Clean C, and by then the infection has looped back through D and E and is sitting on A and B again. We were playing whack-a-mole against an opponent that never got tired, never went home, and worked every machine simultaneously while we worked one at a time.

The arithmetic of losing

The failure was mathematical, not technical. Our cleaning rate was serial: one technician, one machine, some number of minutes. The malware’s infection rate was parallel: every infected machine was a transmitter, all the time. When the propagation rate exceeds the cleaning rate, the infected population grows while you work. You can staff up, and we did, but you are racing an exponential with a linear tool.

Every IT team that has fought a network worm knows this feeling, and most learn the answer the same way we did: the hard way.

We tried the obvious countermove first: more hands. Pull technicians from other work, clean faster. It helped the way bailing helps a holed boat, which is to say it changed the numbers without changing the outcome. Parallel infection beats serial cleaning at any staffing level you can afford, because the malware recruits every machine it captures while each technician remains exactly one technician. The staffing response fails mathematically before it fails operationally, and recognizing that early saves you the week we spent proving it.

Stop treating machines. Treat the network.

The cycle broke when we changed the unit of response. Instead of cleaning machines inside a hostile network, we isolated the workstation network itself, then split the population: a clean segment for disinfected machines, a dirty segment for everything else. Machines moved one direction only, dirty to clean, and only after verification. The transmitters could no longer reach the cured.

Once separation was in place, the math flipped. Every cleaned machine was permanently subtracted from the infected population instead of temporarily. The dirty segment shrank monotonically to zero, and the incident, which had run for a long, demoralizing stretch, ended in an orderly march.

The principle underneath

Containment precedes eradication. That ordering is now standard incident-response doctrine, but doctrine reads like a slide until you have watched a cleaned machine light up again ninety seconds after you left the desk. The lived version of the rule is simpler: never fight an infection on ground the infection controls. Take the ground first.

The same ordering governs modern incidents. Ransomware response starts with isolating segments, not restoring files. Compromise response starts with cutting attacker access, not resetting passwords one account at a time while the attacker holds the mailbox that receives the resets. Every generation of defenders rediscovers the ordering, usually mid-incident.

When executives ask me what makes a security war story worth publishing, this is my example. Not because the malware was exotic. Because the failure mode, serial defense against a parallel attacker, is universal, and a reader who absorbs it will recognize it in incidents that have not happened yet.

For more from this series, see the The Cybersecurity Hub: breaches, audits, and hard-won security lessons from four decades in the trenches.

Frequently Asked Questions