My disaster recovery education started with lightning. Early eighties, a small company where I was working overnight in the computer room because I liked the cold air, a minicomputer humming, rain starting outside. Then lightning hit the power pole directly outside the building and the room lit up: sparks, and every machine in it fried. Not crashed, fried. I was not touching anything metal, which is why I am here to describe it.
The company survived because of two unglamorous facts. I had backups of everything, because I am smart about exactly one thing, and the business could operate on paper for the weeks it took to get replacement hardware installed. That was my first complete disaster recovery event, and it contained the whole discipline in miniature: you cannot prevent the lightning; you can only decide, in advance, what survives it.
The DR site and the backhoe
Years later, running infrastructure for a much larger company, I built a real disaster recovery site: another facility roughly forty miles away, which was a serious distance in the days before the internet, connected first by a pair of T1 lines and later a T3. And I learned the lesson every DR site owner learns: the site is only as real as its link. A backhoe somewhere along the route cut the T3 once. Lightning strikes on the transmission poles damaged it other times. Each cut quietly converted our disaster site into an expensive building full of idle equipment, and one of those outages overlapped the worst crash of my career, when the DR site we needed was unreachable for the most mundane reason imaginable: the phone company had not come out yet.
A DR capability has a heartbeat, and the heartbeat is the link plus the currency of the data on the far side. Monitor either one casually and you own a theatrical set.
The tabletop with the truck bomb
The best training I ever received was a tabletop exercise at a disaster recovery conference in San Diego. Around a hundred and fifty of us in a room, wall screens playing fabricated news coverage, and a scenario with teeth: a truck bomb detonated at our fictional office, with an active shooter inside the building. Two simultaneous disasters, which is how the real ones like to arrive. Tables were assigned roles; mine was the decision-making committee, so my job was to decide, fast, with incomplete information.
The revelation was not tactical, it was structural. As our table issued decisions, the government officials present kept responding with the same sentence: we have to send that up the line, we cannot decide that at our level. Different organizations at the same table had completely different shapes of authority, and some of them could not act at all inside the timeline the scenario allowed. You do not discover that reading your DR binder. You discover it when the fake news is playing and your mutual-aid partner turns out to need three approvals to open a door. I recommend serious tabletop exercises to anyone who owns security or recovery responsibility; you will learn more about your organization in three hours than in a year of plan reviews.
You cannot prevent the lightning. You can only decide, in advance, what survives it.Share on X
The plan that history tested
I once attended a speech by the woman who ran disaster recovery for the World Trade Center trading operations, under a legal requirement to be back up within a day. Her plan was elaborate to the point of buses, alternate personnel rotations, the works, and it had struck some people as excessive. Then September 11th came, and the plan ran for real, and it worked. Her one unplanned failure: the collapse landed on the area’s main telephone switch, and communications, the thing every other part of the plan silently assumed, went spotty across the region. They worked around it, but the lesson stayed with me. Even the best-tested plan carries an assumption it never noticed making, and it is usually about communications.
I later went through CERT, the Community Emergency Response Team program, twice, seven three-hour sessions of lectures and live drills each time: triage on simulated casualties, fire suppression with real extinguishers, which is nothing like you imagine it. It ranks among the more fulfilling things I have done, and it closed a loop for me: disaster recovery is one discipline that happens to have a computer-room department.
What do twenty years of disaster recovery distill to?
Assume the disaster; choose in advance what survives it. Verify the safety nets on a schedule, because they fail silently and in groups. Exercise the plan against scenarios with teeth, including the authority structures, not just the technology. And interrogate your communications assumption, because that is where the plan you tested still hides the failure you did not. The same principles run inside every transformation I write about, and they scale all the way down to the tested backups, spare machine, and hotspot-ready laptop I keep at home. The lightning does not check the size of the building.
For more from this series, see the The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
The Guides That Get Your Book Written, Published, and Sold
Four short, practical guides on writing, publishing, and selling your book, plus the occasional note when there's something worth your time. No fluff, no daily inbox clutter. Drop your email and they're yours.
We use MailerLite to manage our list and send these emails. Your address is used only to send you what you signed up for. We will not sell it, share it, or use it for anything else, and you can unsubscribe anytime.