Twenty Years of Disaster Recovery: What the Real Ones Taught Me

My disaster recovery education started with lightning. Early eighties, a small company where I was working overnight in the computer room because I liked the cold air, a minicomputer humming, rain starting outside. Then lightning hit the power pole directly outside the building and the room lit up: sparks, and every machine in it fried. Not crashed, fried. I wasn’t touching anything metal, and that’s why I’m here to describe it.

The company survived because of two unglamorous facts.

I had backups of everything, because I’m smart about exactly one thing. The business could operate on paper for the weeks it took to get replacement hardware installed. That was my first complete disaster recovery event, and it held the whole discipline in miniature. You can’t prevent the lightning. You can only decide, in advance, what survives it.

Plenty of organizations still haven’t learned that. On July 19, 2024, the day of the CrowdStrike outage, I was at the doctor’s, and the clinic was completely down. No records, no access to anything, and even the little kiosk didn’t work. This is Florida. We have hurricanes, and the internet is guaranteed to go down. An airline that goes down loses money, but a hospital could lose people. They should have dropped to a degraded mode. They had nothing.

The DR site and the backhoe

Years later, running infrastructure for a much larger company, I built a real disaster recovery site. Another facility roughly forty miles away. That was a serious distance in the days before the internet, connected first by a pair of T1 lines and later a T3. And I learned the lesson every DR site owner learns: the site is only as real as its link. A backhoe somewhere along the route cut the T3 once. Lightning strikes on the transmission poles damaged it other times.

Each cut converted our disaster site into an expensive building full of idle equipment. One of those outages overlapped the worst crash of my career, when the DR site we needed was unreachable for the most mundane reason imaginable: the phone company hadn’t come out yet.

A DR capability has a heartbeat: the link, plus the currency of the data on the far side. A recovery site nobody monitors is a cheat, because the money buys a feeling of safety and nothing else. If nobody watches the link and the data every day, you own an expensive stage set, and you’ll find out on the day you need it.

The tabletop with the truck bomb

The best training I ever received was a tabletop exercise at a disaster recovery conference in San Diego.

Around a hundred and fifty of us in a room, with wall screens playing fabricated news coverage. The scenario had teeth. A truck bomb detonated at our fictional office, with an active shooter inside the building. Two simultaneous disasters, the way the real ones like to arrive. Tables were assigned roles; mine was the decision-making committee, so my job was to decide, fast, with incomplete information.

The revelation wasn’t tactical, it was structural. As our table issued decisions, the government officials present kept responding with the same sentence: we have to send that up the line, we can’t decide that at our level. Different organizations at the same table had completely different shapes of authority, and some of them couldn’t act at all inside the timeline the scenario allowed. You don’t discover that reading your DR binder.

You discover it when the fake news is playing and your mutual-aid partner turns out to need three approvals to open a door. I recommend serious tabletop exercises to anyone who owns security or recovery responsibility; you’ll learn more about your organization in three hours than in a year of plan reviews.

The plan that history tested

I once attended a speech by the woman who ran disaster recovery for the World Trade Center trading operations, under a legal requirement to be back up within a day. Her plan was elaborate to the point of buses, alternate personnel rotations, the works, and it had struck some people as excessive. Then September 11th came, and the plan ran for real, and it worked.

Her one unplanned failure: the collapse landed on the area’s main telephone switch, and communications, the thing every other part of the plan silently assumed, went spotty across the region. They worked around it, but the lesson stayed with me. Even the best-tested plan carries an assumption it never noticed making, and it’s usually about communications.

People who call a plan like hers excessive are judging it on a normal Tuesday. I’d much prefer defending an overbuilt plan to a budget committee over explaining a missing one to employees who can’t get paid on Friday.

I was in charge of disaster recovery and business continuity at the last three companies I worked for, along with the network guy, who was my peer. The two of us were always thrilled when a disaster hit, because a real failure made the case for us. We had a failure? Great. Now we can get a new generator.

I later went through CERT, the Community Emergency Response Team program, twice, seven three-hour sessions of lectures and live drills each time: triage on simulated casualties, fire suppression with real extinguishers. That’s nothing like you imagine it. It ranks among the more fulfilling things I’ve done, and it closed a loop for me: disaster recovery is one discipline that happens to have a computer-room department.

What do twenty years of disaster recovery distill to?

Assume the disaster; choose in advance what survives it.

Verify the safety nets on a schedule, because they fail silently and in groups. Exercise the plan against scenarios with teeth, including the authority structures as well as the technology. And interrogate your communications assumption, because even a plan you’ve tested can hide an untested failure there. The same principles run inside every transformation I write about, and they scale all the way down to the tested backups, spare machine, and hotspot-ready laptop I keep at home.

After twenty years of this, what frustrates me most is the executive who signs the DR budget once and never asks to see it work. The lightning won’t care how big the building is or how busy everyone was that quarter. It’ll find the net nobody checked, and the people who lose their paychecks or their records never had a say in skipping the test.

For more from this series, see The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.

Frequently Asked Questions