Table of Contents
TL;DR
In one morning we lost the power, the generator, the UPS coverage, the cache battery, the link to the disaster site, and the tape backup. Six independent safety nets, six failures. I ran disaster recovery for twenty years, and this was the day that taught me the discipline. The company came back because one consultant rebuilt a disk mapping table by hand, from memory, and afterward I tore the guts out of that computer room and rebuilt everything redundant. The deeper lesson: I had ordered full redundancy years earlier, and nobody told me one cabinet did not have it.
I was a hundred and thirteen miles from the office, driving north to a Renaissance fair with a camera on the seat beside me, when my consultant called. Three friends in full pirate costume were waiting for me at the fairgrounds. This was going to be a great day.
“Richard,” he said, “you do know about the crash.”
I did not know about the crash. What followed was a phone call I have retold for decades, because each exchange in it removed a safety net I believed we had, and by the end there were none left.
The cascade, one net at a time
The power had failed in the area around the data center. That should not have mattered: we had a generator. The generator had not come on, because the transfer switch, the device whose one job is to move the building onto generator power, had failed. That should not have mattered either: the equipment was on UPS units, which would carry the load long enough for clean shutdowns.
There were three UPS units. We needed four.
The cabinet holding roughly five hundred disk drives, the storage heart of the operation, was supposed to be cross-plugged across redundant UPS units. Years earlier I had given the standing order: dual redundancy on everything, two power paths, two of every path that mattered. The installers had not done it for that cabinet, there was no fourth unit, the cross-plugging went to units already loaded, and nobody had told me. The cabinet lost power without protection.
Still survivable, in theory. When power returned, the cabinet should have restarted. Except a component the size of a quarter, the little coin battery whose only job is to hold the cabinet’s cache alive for the few seconds it takes to flush to disk, had failed too. The cache died unflushed, and it took with it the master mapping table, the structure that described how five hundred physical drives assembled into the mirrored volumes the business ran on. The drives were fine. The knowledge of what they were was gone.
“Fail over to the DR site,” I said. The T3 line to the disaster recovery site, forty miles away, had gone down two days earlier, and the phone company had not yet come out to repair it. “Then we restore from tape.” The tape backup from just before the crash had errored out. There was no tape.
Power, generator, UPS, cache battery, DR link, backup. Six layers, engineered precisely so that no single failure could hurt us. All six.
The rebuild from memory
By then I was at the fair, and I made the call every leader of a remote crisis has to make honestly: a hundred and fifteen miles away, I was more useful on the phone than on the road. We were dead in the water either way. Hours of attempts went nowhere, and then my consultant called with the strangest request of my career: permission to try something he refused to describe.
“I’m not going to tell you. Just give me permission to do it.” We had nothing left to lose, and I said so, and I said do it.
A couple of hours later he called back: he had rebuilt the disk mapping table by hand. From memory. The table that described how five hundred drives fit together, reconstructed keystroke by keystroke by a man who had internalized the configuration deeply enough to recreate it without documentation. We were down roughly eight hours end to end, and the remainder of the recovery was me walking teams through restarting applications that had never been designed to survive a crash, which is its own quiet indictment, told in a companion article.
Power, generator, UPS, cache battery, DR link, tape backup. Six safety nets engineered so no single failure could hurt us. All six failed in one morning.Share on X
What actually failed
Here is the thing: the cascade was not six pieces of bad luck. It was one management failure wearing six costumes. The transfer switch had presumably never been tested under real failure. The missing fourth UPS was an instruction given and never verified. The coin battery was a consumable nobody was replacing on schedule. The T3 had been down for two days, which means the DR site had been fiction for two days and no alarm treated that as an emergency. The tape failure was discovered at restore time, which means restores were never being tested. Every layer had failed silently, in advance, and the power outage merely conducted the audit.
So the lesson I took was not “have more redundancy,” although we did that too, gutting and rebuilding the entire computer room afterward with genuinely redundant everything. The lesson was check and verify. An instruction is not a control. “I ordered dual redundancy” and “we have dual redundancy” are different sentences, and the distance between them was eight hours of downtime and one irreplaceable consultant’s memory. After that, I verified, personally and on a schedule, that what I had ordered existed, and we never had that class of problem again.
Every executive who believes their infrastructure is redundant should sit with this story for a minute, then ask one question at the next operations review: when did we last prove it, layer by layer, under failure? Not audit the paperwork. Prove it. Somewhere in your building is a cabinet nobody told you about.
For more from this series, see the The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
The Guides That Get Your Book Written, Published, and Sold
Four short, practical guides on writing, publishing, and selling your book, plus the occasional note when there's something worth your time. No fluff, no daily inbox clutter. Drop your email and they're yours.
We use MailerLite to manage our list and send these emails. Your address is used only to send you what you signed up for. We will not sell it, share it, or use it for anything else, and you can unsubscribe anytime.