The Virus That Crashed Our Own Email Server

Before ransomware, before phishing kits, there was ILOVEYOU, one of the first destructive mass-mailing worms. When it hit us, it hit every system, but its real damage wasn’t to the machines it infected. Its damage was volume. The worm mailed itself to everyone in every address book it touched, and it did that from hundreds of machines at once.

We were running an early version of Exchange, and it choked. The flood of outbound mail was so large that the server ran out of disk space and crashed. Our own email killed our email system.

That one was entirely avoidable. The worm did nothing clever. It used address books and a mail server exactly the way they were built to be used, only faster than any person could, and that was enough to cut a company off from itself. Any system that its own normal traffic can kill has a design problem, and every IT team should know what its systems do when ordinary use suddenly runs a thousand times faster.

What is sneakernet?

With email down company-wide, coordination reverted to what we used to call sneakernet: put on your sneakers and walk to the person you need.

Every question that would have been a two-line email became a trip across the building. The entire company slowed to the speed of hallway conversations, and you don’t appreciate what that means until it happens. Anything that couldn’t wait for a walk simply waited, and the business stalled. Anyone who calls an email outage an IT inconvenience hasn’t lived through one.

That experience permanently changed how I think about email. For most companies it’s the nervous system, and when the nervous system goes down, the organization discovers it can’t move. Executives should budget for it and protect it that way. A company that treats email as one tool among many will staff it that way too, right up until the day it learns otherwise.

No playbook, no precedent

What’s hard to convey now is how alone you were in an outbreak then. There was no threat-intelligence feed pushing indicators to your dashboard, no vendor bulletin arriving ahead of the malware, no incident-response retainer to call. Antivirus signatures for a new worm lagged its spread by days, which during ILOVEYOU meant the entire event. You diagnosed by symptom: the mail queues exploding, the same subject line multiplying, users calling about a message from a colleague that seemed off.

We were writing the playbook while the building burned, and so was every other company on the planet that week. I keep that memory around as a calibration. Modern security teams drown in intelligence and sometimes envy a simpler era. The simpler era was worse. Information asymmetry favored the worm completely, and the defenders’ only real-time network was the telephone.

It makes me impatient when a security team complains that it has too much intelligence to read. Too much information is a management problem. Having none, with the worm already in your mail queues, is a helpless feeling I wouldn’t wish on anyone who runs a network.

How long did recovering from the email virus take?

Recovery took weeks, and it came in ugly layers. First, we had to figure out what was happening at all, in an era when there was no threat-intel feed telling you a worm was circling the planet. Then we had to stop the reinfection cycle: clean one machine and its infected neighbors would reinfect it before the technician reached the next desk.

The eventual answer was segmentation used as a treatment instead of a defense: we created a clean subnet and moved disinfected machines into it, so the sick population couldn’t touch the cured one. The infected side shrank machine by machine until it was gone.

The server itself was its own project. We didn’t restore from backup. We went into the Exchange queues and purged them by hand, a procedure we first had to research and then perform on a production system holding the company’s mail. Delete the worm’s traffic, preserve the legitimate mail, free the disks, bring the server back. Then watch it, because if one infected machine remained, the flood would resume.

Purging queues by hand on a live server is the kind of work that keeps an operations person awake. One wrong delete destroys legitimate company mail nobody can recover, and nobody thanks you for the thousands you got right. The people who do that work under that pressure deserve far more credit than most companies ever give them.

What does ILOVEYOU still teach?

First: availability is a security property. The worm encrypted nothing and stole nothing from us, and we still lost weeks, because the attack consumed a resource. Modern denial-of-service thinking starts exactly here. Second: reinfection is the real enemy of cleanup. Any recovery plan that cleans machines one at a time inside a connected population is a plan to clean the same machines repeatedly. You separate clean from dirty first, then you cure.

Third: have a fallback channel. Every incident-response plan that assumes email works should be assumed broken. Ours was walking. Yours should be better than walking. For more from this series, see The Cybersecurity Hub: breaches, audits, and hard-won security lessons from 33 years in the trenches.

Frequently Asked Questions