Disaster Recovery Is Part of the Transformation, or It Is Not One

Disaster recovery was one of my responsibilities at the major national retailer, and I held it through both transformations.

That gave me a vantage point most transformation stories lack. We had some disasters. Real ones, the kind that convert your DR documentation from a compliance artifact into a set of claims about to be tested. Some claims held. Some taught us things, including the discovery of a critical server under a desk in accounting that no plan had ever heard of.

I care about this more than almost anything else in a transformation, because I’ve lived through the day a recovery plan had to work and didn’t. Transformation teams get the credit for the new platforms. The people who inherit the recovery get the early-morning phone call, and nothing on the project plan tells them what to do next.

Recovery is a property of the environment

The disasters drilled one principle into me. A disaster recovery plan is a precise set of claims about a specific environment: these systems, on this hardware, with these dependencies, restored in this order, from these backups, within this time. Change the environment and the claims silently expire, while the plan keeps looking exactly as authoritative on the shelf.

Now consider what a transformation is: the deliberate, wholesale changing of the environment. Every system we moved to the new platform, every function we lifted to SaaS, every integration we rebuilt was a page of the DR plan invalidated. A transformation is, among everything else it is, the largest disaster-recovery-plan destruction event a company ever voluntarily runs.

That forces the choice in the title. Either recovery is transformed alongside the systems, plans rewritten, backup architectures rebuilt for the new platforms, restoration sequences re-derived from the new dependencies, or the company exits the transformation more modern and less recoverable than it entered, holding documentation for an environment that no longer exists.

I’ve got little sympathy for a company that walks out of a transformation less recoverable than it walked in. It chose that, one deferred workstream at a time, and the bill arrives on a day when customers, stores and payroll are all waiting on systems that won’t come back.

What transforming recovery looked like

In our program, DR work rode inside the transformation instead of trailing it. When a function moved, its recovery moved as part of the same effort: new backup coverage before old coverage lapsed, restoration procedures rewritten against the new platform, dependencies re-mapped because the new world’s restoration order wasn’t the old world’s.

The SaaS moves changed the recovery question entirely, from “how do we restore this” to “what does the vendor guarantee and what’s our plan when the vendor is the disaster,” which is a different plan, needing different thinking, and it doesn’t write itself.

And we tested, because the disasters had cured us of trusting untested claims. A recovery plan that’s never run is a hypothesis with a binder. Some of ours failed their first tests, and that’s the point of testing: failures found in an exercise cost embarrassment, and failures found in a disaster cost the company.

One of those disasters still defines the job for me. A consultant called early one morning about weird errors on one of the two mirrored disks holding the operating system for our SAN cabinets, and asked to replace it. It was mirrored, so I figured, what the fuck, replace it. Within about an hour we knew we were in serious trouble. Pulling the disk triggered a bug in the firmware and cluster software, the SQL database was corrupt, the DR site was down and the backups were bad.

Why is testing where recovery becomes real?

A word on what “tested” means, because people stretch the term. Reading the plan in a conference room is a review, and it catches missing pages. A tabletop exercise, walking the team through a scenario hour by hour, is better, and it catches wrong assumptions: the restore step that depends on a person who would be evacuated, the vendor contact who left two years ago.

But the tests that changed our plans were the ones that touched systems: restoring from the backups, failing over, on a schedule, against a clock. Those are the tests that discover the backup that completes nightly and restores nothing usable, and that discovery is only survivable when it happens in an exercise. (See also why the AWS Bahrain customers needed copies elsewhere.)

Ours didn’t happen in an exercise. We were down 72 hours, almost to the minute, and a lot of that went to waiting on backups, because my backup person was on vacation in Acapulco and we couldn’t reach him at first. Elvis, my DBA, rebuilt the database in what looked to me like magic, and we lost an hour of data and spent months recovering it. When it was over I went straight to the Renaissance Faire, paid for about four hour-long massages, and was too tired to sleep.

Post-transformation, the first live restoration test of each migrated system is the moment its recovery stops being a document and becomes a proven capability. We scheduled those tests as part of each migration’s closure, so no system was declared transformed while its recovery remained a hypothesis. I’d hold any transformation team to the same rule today.

What should executives ask about disaster recovery?

If your company is mid-transformation or recently finished one, ask this in your next steering meeting: which of our disaster recovery plans have been rewritten and tested against the new environment, and which still describe the old one? Expect an uncomfortable answer. DR is the workstream most easily deferred, it produces nothing visible when things go well, and nobody finds out it’s missing until the day the systems go down.

The transformation books executives write almost never contain a recovery chapter, and I read that as a sign of how rarely the authors carried that responsibility. That’s a gap, because the people who did carry it know things the rest of the field needs to hear. If you’ve stood in a dead computer room waiting on tapes, put that in your book. Your competitors can’t fake it, and the readers who’ve been there will trust every other page because of it.

For more from this series, see The Digital Transformation Hub: real transformations, lived from the inside, decades before the term existed.

Frequently Asked Questions