I was a hundred and thirteen miles from the office, driving north to a Renaissance fair with a camera on the seat beside me, when my consultant called. Three friends in full pirate costume were waiting for me at the fairgrounds. This was going to be a great day. It was the Kearney Park Renaissance Faire in Fresno.
My three friends in pirate costume at the Kearney Park faire in Fresno, the day the computers went down. Copyright © Richard Lowe.
“Richard,” he said, “you do know about the crash.”
I didn’t know about the crash. What followed was a phone call I have retold for decades. Each exchange in it removed a safety net I believed we had. By the end there were none left.
How did each safety net fail in the cascade?
Each layer failed silently in advance, and the outage exposed it.
The power had failed in the area around the data center.
That shouldn’t have mattered: we had a generator. The generator hadn’t come on, because the transfer switch, the device whose one job is to move the building onto generator power, had failed. That shouldn’t have mattered either: the equipment was on UPS units. That would carry the load long enough for clean shutdowns.
There were three UPS units. We needed four.
The cabinet holding roughly five hundred disk drives, the storage heart of the operation, was supposed to be cross-plugged across redundant UPS units. Years earlier I’d given the standing order: dual redundancy on everything, two power paths, two of every path that mattered. The installers hadn’t done it for that cabinet, there was no fourth unit, the cross-plugging went to units already loaded, and nobody had told me. The cabinet lost power without protection.
Still survivable, in theory. When power returned, the cabinet should have restarted. Except one component the size of a quarter had failed too. The little coin battery whose only job is holding the cabinet’s cache alive for the few seconds it takes to flush to disk. The cache died unflushed, and it took with it the master mapping table, the structure that described how five hundred physical drives assembled into the mirrored volumes the business ran on. The drives were fine. The knowledge of what they were was gone.
“Fail over to the DR site,” I said. The T3 line to the disaster recovery site, forty miles away, had gone down two days earlier, and the phone company hadn’t yet come out to repair it. “Then we restore from tape.” The tape backup from just before the crash had errored out. There was no tape.
A server rack with its tape drives. A tape backup proves itself only at restore time. Copyright © Richard Lowe.
Power, generator, UPS, cache battery, DR link, backup. Six layers, engineered so that no single failure could hurt us, and every one of them had failed. We’d spent years building that protection, and on the one day it mattered, none of it was there.
How was the system rebuilt from memory?
A tape backup proves itself only at restore time.
By then I was at the fair, and I made the call every leader of a remote crisis has to make: a hundred and thirteen miles away, I was more useful on the phone than on the road. We were dead in the water either way. Hours of attempts went nowhere, and then my consultant called with the strangest request of my career: permission to try something he refused to describe.
“I’m not going to tell you. Just give me permission to do it.” We had nothing left to lose, and I said so, and I said do it.
A couple of hours later he called back: he’d rebuilt the disk mapping table by hand. From memory. The table that described how five hundred drives fit together, reconstructed keystroke by keystroke by a man who had internalized the configuration deeply enough to recreate it without documentation. We were down roughly eight hours end to end, and the remainder of the recovery was me walking teams through restarting applications nobody had designed to survive a crash. That’s its own quiet indictment, told in a companion article.
My take
Verify your redundancy with your own eyes. We had six safety nets, power, generator, UPS, cache battery, DR link and tape, and every one failed the same morning. I’d ordered full redundancy years earlier, and nobody told me one cabinet never got it. A standing order means nothing until somebody checks it. Walk the cabinets, test the restores, and chase the dead link the day it drops. A consultant rebuilding a mapping table from memory is a miracle, and you can’t plan on miracles.
Power, generator, UPS, cache battery, DR link, tape backup. Six safety nets engineered so no single failure could hurt us. All six failed in one morning.Share on X
What failed in the cascading disaster?
The cascade was one management failure wearing six costumes, and I include myself in that. Presumably nobody had ever tested the transfer switch under real failure. The missing fourth UPS was an instruction given and never verified. The coin battery was a consumable nobody was replacing on schedule. The T3 had been down for two days, so the DR site had been fiction for two days and no alarm treated that as an emergency.
We discovered the tape failure at restore time. That means nobody was ever testing restores. Every layer had failed silently, in advance, and the power outage merely conducted the audit.
So the lesson I took wasn’t “have more redundancy,” although we did that too, gutting and rebuilding the entire computer room afterward with redundant everything.
The lesson was check and verify. An instruction isn’t a control. “I ordered dual redundancy” and “we have dual redundancy” are different sentences, and the distance between them was eight hours of downtime and one irreplaceable consultant’s memory. After that, I verified, personally and on a schedule, that what I had ordered existed, and we never had that class of problem again.
| Safety net | Underlying failure | What to verify |
|---|---|---|
| Generator transfer switch | Never tested under real failure | Test transfer switches and failover under real conditions |
| UPS coverage | Fourth unit ordered, never installed, nobody checked | Walk the cabinets and confirm what was ordered exists |
| Cache coin battery | A consumable nobody replaced on schedule | Replace consumable components on schedule |
| T3 link to DR site | Down two days with no alarm treating it as urgent | Treat a down DR link as an emergency the day it drops |
| Tape backup | Errored out; restores were never tested | Regularly prove that backups restore |
Every executive who believes their infrastructure is redundant should take this story into the next operations review and ask one question: when did we last prove it, layer by layer, under failure? Reading the paperwork doesn’t count. An executive who calls redundancy a solved problem because a purchase order says so is being reckless, because the people who pay for that belief are the operations staff working the phones and the business sitting dead in the water while they do.
The same lesson played out on a much bigger stage in October 2026. After Ukrainian drones hit three of Yandex’s five data centers, Geopolitical Futures reported that the company’s thin backup systems delayed the recovery of its financial operations. Its two surviving data centers were left carrying the entire load. The government had been urging big Russian tech companies to improve fault tolerance for over a year. A recovery plan only counts if it survives the disaster it was written for.
For more from this series, see The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
