Table of Contents
TL;DR
The hardest part of the worst crash of my career was not the hardware. It was the applications, because none of them had been designed to survive dying. After a cascading failure took us down for eight hours, restoring the disks was only the midpoint. I spent the rest of the day walking teams through restarting applications that assumed they would never stop: unknown states, no restart procedures, no design for recovery at all. Resilience is an application property, and most software is written as if crashes happen to other programs.
In the story of the worst crash of my career, the drama belongs to the infrastructure: six failed safety nets and a consultant rebuilding a disk mapping table from memory. I promised the epilogue would get its own article, and this is it, because the epilogue contained a lesson bigger than the crash. Once the disks were back, the applications had to come up. They were not designed to.
Not “the restart was slow.” Not designed to. The applications had been written on the working assumption that they would never stop, and when that assumption failed, every one of them turned into a puzzle. I spent the rest of that day on the phone, walking people through bringing the business back one application at a time, improvising procedures that should have existed in a binder.
What “not designed to recover” looks like
If you have never lived it, here is the texture. An application dies mid-flight, and its data freezes in a state its authors never imagined: transactions half-committed, work files half-written, flags set that only make sense if the next step, which never ran, had run. Start it again and it does not resume; it walks into that frozen state and behaves in ways nobody can predict, because nobody ever asked the question. So before anything restarts, humans have to reconstruct what the application was doing at the instant of death and manually put the data back into a condition the software recognizes. Multiply that by every application in the company, sequence it by their dependencies on each other, which are also documented nowhere, and you understand why the disks coming back was the midpoint of the outage rather than the end.
The disks came back in hours. The applications took the rest of the day, because nobody had ever asked what they should do after dying.Share on X
Why software gets written this way
No programmer decides to make an unrecoverable application. The omission is structural. Applications get specified by what they do, never by how they die: the requirements say process the orders, print the reports, post the transactions, and recovery appears nowhere because the people writing requirements are describing the business working, not the business resuming. Then the software runs for years without a serious crash, which reads as evidence the question never needed asking, right up until the morning it does. Our applications came from a consulting firm that built exactly what was specified. Nobody specified surviving.
The infrastructure world learned this lesson formally and built it into everything: databases journal their transactions, file systems log their intents, storage arrays flush caches on power loss, all of it designed around the assumption of sudden death. The application layer, sitting on top of all that engineered resilience, routinely assumes immortality. The stack recovers beautifully up to the exact boundary where the business logic begins.
What I did about it
After that crash, recovery became a requirement in my shop rather than a hope. Every application, existing or new, needed an answer to three questions, written down and tested. What state can this application be in when it dies mid-work? What is the procedure, automated or manual, for bringing it from that state to a clean start? And what must come up before it, and after it, in the dependency order of the whole environment? The third question produced the most valuable single document to come out of the entire incident: the restart sequence for the business, which had previously existed only as fragments in various heads, the same way the disk configuration had existed only in my consultant’s.
None of this required rewriting the software, though some applications eventually earned that. Most of the gap closed with procedures: documented, rehearsed restart runbooks that converted each application from a puzzle into a checklist. The distinction matters for anyone inheriting a legacy environment, because “our applications cannot survive a crash” sounds like a rebuild mandate, and it is usually a documentation mandate wearing scarier clothes.
The question for executives
Your infrastructure people can likely tell you their recovery time for servers, storage, and networks, because the infrastructure world measures such things. Ask instead: if every application stopped mid-transaction right now, who knows the order and the procedures for bringing the business logic back, and where is that written? In most companies the honest answer is a list of names, and names retire, take vacations, and, as my crash taught me twice in one day, occasionally hold the only copy of something the company cannot operate without. The applications are the business. Their recovery deserves at least the engineering attention the plumbing under them gets.
For more from this series, see the The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
The Guides That Get Your Book Written, Published, and Sold
Four short, practical guides on writing, publishing, and selling your book, plus the occasional note when there's something worth your time. No fluff, no daily inbox clutter. Drop your email and they're yours.
We use MailerLite to manage our list and send these emails. Your address is used only to send you what you signed up for. We will not sell it, share it, or use it for anything else, and you can unsubscribe anytime.