My disaster recovery education started with lightning. Early eighties, a small company where I was working overnight in the computer room because I liked the cold air, a minicomputer humming, rain starting outside. Then lightning hit the power pole directly outside the building and the room lit up: sparks, and every machine in it fried. Not crashed, fried. I wasn’t touching anything metal, and that’s why I’m here to describe it.
The company survived because of two unglamorous facts. I’d backups of everything, because I’m smart about exactly one thing. The business could operate on paper for the weeks it took to get replacement hardware installed. That was my first complete disaster recovery event, and it held the whole discipline in miniature. You can’t prevent the lightning. You can only decide, in advance, what survives it.
The DR site and the backhoe
Years later, running infrastructure for a much larger company, I built a real disaster recovery site. Another facility roughly forty miles away. That was a serious distance in the days before the internet, connected first by a pair of T1 lines and later a T3. And I learned the lesson every DR site owner learns: the site is only as real as its link. A backhoe somewhere along the route cut the T3 once. Lightning strikes on the transmission poles damaged it other times.
Each cut quietly converted our disaster site into an expensive building full of idle equipment. One of those outages overlapped the worst crash of my career, when the DR site we needed was unreachable for the most mundane reason imaginable: the phone company hadn’t come out yet.
A DR capability has a heartbeat, and the heartbeat is the link plus the currency of the data on the far side. Monitor either one casually and you own a theatrical set.
The tabletop with the truck bomb
The best training I ever received was a tabletop exercise at a disaster recovery conference in San Diego. Around a hundred and fifty of us in a room, with wall screens playing fabricated news coverage. The scenario had teeth. A truck bomb detonated at our fictional office, with an active shooter inside the building. Two simultaneous disasters, the way the real ones like to arrive. Tables were assigned roles; mine was the decision-making committee, so my job was to decide, fast, with incomplete information.
The revelation wasn’t tactical, it was structural. As our table issued decisions, the government officials present kept responding with the same sentence: we have to send that up the line, we can’t decide that at our level. Different organizations at the same table had completely different shapes of authority, and some of them couldn’t act at all inside the timeline the scenario allowed. You don’t discover that reading your DR binder.
You discover it when the fake news is playing and your mutual-aid partner turns out to need three approvals to open a door. I recommend serious tabletop exercises to anyone who owns security or recovery responsibility; you’ll learn more about your organization in three hours than in a year of plan reviews.
You can’t prevent the lightning. You can only decide, in advance, what survives it.Share on X
The plan that history tested
I once attended a speech by the woman who ran disaster recovery for the World Trade Center trading operations, under a legal requirement to be back up within a day. Her plan was elaborate to the point of buses, alternate personnel rotations, the works, and it had struck some people as excessive. Then September 11th came, and the plan ran for real, and it worked.
Her one unplanned failure: the collapse landed on the area’s main telephone switch, and communications, the thing every other part of the plan silently assumed, went spotty across the region. They worked around it, but the lesson stayed with me. Even the best-tested plan carries an assumption it never noticed making, and it’s usually about communications.
I later went through CERT, the Community Emergency Response Team program, twice, seven three-hour sessions of lectures and live drills each time: triage on simulated casualties, fire suppression with real extinguishers. That’s nothing like you imagine it. It ranks among the more fulfilling things I’ve done, and it closed a loop for me: disaster recovery is one discipline that happens to have a computer-room department.
What do twenty years of disaster recovery distill to?
Assume the disaster; choose in advance what survives it. Verify the safety nets on a schedule, because they fail silently and in groups. Exercise the plan against scenarios with teeth, including the authority structures, not just the technology. And interrogate your communications assumption, because that’s where the plan you tested still hides the failure you didn’t. The same principles run inside every transformation I write about, and they scale all the way down to the tested backups, spare machine, and hotspot-ready laptop I keep at home.
The lightning doesn’t check the size of the building.
For more from this series, see The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
