In the story of the worst crash of my career, the drama belongs to the infrastructure: six failed safety nets and a consultant rebuilding a disk mapping table from memory.
I promised the epilogue would get its own article, and this is it, because the epilogue contained a lesson bigger than the crash. Once the disks were back, the applications had to come up. They weren’t designed to.
Not “the restart was slow.” Not designed to. The applications had been written on the working assumption that they’d never stop, and when that assumption failed, every one of them turned into a puzzle. I spent the rest of that day on the phone, walking people through bringing the business back one application at a time, improvising procedures that should have existed in a binder.
What “not designed to recover” looks like
If you’ve never lived it, here’s the picture. An application dies mid-flight and its data freezes in a state its authors never imagined. Transactions half-committed. Work files half-written. Flags set that only make sense if the next step, which never ran, had run. Start it again and it doesn’t resume; it walks into that frozen state and behaves in ways nobody can predict, because nobody ever asked the question.
So before anything restarts, humans have to reconstruct what the application was doing at the instant of death and manually put the data back into a condition the software recognizes. Multiply that by every application in the company, sequence it by their dependencies on each other, which are also documented nowhere, and you understand why the disks coming back was the midpoint of the outage instead of the end.
That part of an outage is worse than the hardware part. A failed disk tells you what it is. A frozen application makes a room full of exhausted people guess, and every wrong guess can corrupt the data the business needs to open its doors in the morning.
The disks came back in hours. The applications took the rest of the day, because nobody had ever asked what they should do after dying.Share on X
Why does software get written without recovery in mind?
No programmer decides to make an unrecoverable application. The omission is structural. Applications get specified by what they do, never by how they die. The requirements say process the orders, print the reports, post the transactions. Recovery appears nowhere because the people writing requirements are describing the business working, and nobody describes the business resuming. Then the software runs for years without a serious crash, which reads as evidence the question never needed asking, right up until the morning it does.
Our applications came from a consulting firm that built exactly what was specified. Nobody specified surviving.
The infrastructure world learned this lesson formally and built it into everything: databases journal their transactions, file systems log their intents, storage arrays flush caches on power loss, all of it designed around the assumption of sudden death. The application layer, sitting on top of all that engineered resilience, routinely assumes immortality. The stack recovers beautifully up to the exact boundary where the business logic begins.
Applications get specified by what they do, never by how they die.
Those engineered layers have limits too. A different crash kept us down 72 hours, almost to the minute. A consultant wanted to replace a mirrored disk throwing weird errors, and I figured what the fuck, replace the damn thing. Pulling it tripped a bug in the firmware and cluster software, and the SQL database came back corrupt because it couldn’t write its buffers out. Elvis, my DBA, rebuilt it in a way that looked like magic to me, and we still spent months recovering the hour of data we lost.
What I did about it
After that crash, recovery became a requirement in my shop instead of a hope. Every application, existing or new, needed an answer to three questions, written down and tested. What state can this application be in when it dies mid-work? What’s the procedure, automated or manual, for bringing it from that state to a clean start? And what must come up before it, and after it, in the dependency order of the whole environment?
The third question produced the most valuable single document to come out of the entire incident: the restart sequence for the business, which had previously existed only as fragments in various heads, the same way the disk configuration had existed only in my consultant’s.
None of this required rewriting the software, though some applications eventually needed that. Most of the gap closed with procedures: documented, rehearsed restart runbooks that converted each application from a puzzle into a checklist. The distinction matters for anyone inheriting a legacy environment, because “our applications cannot survive a crash” sounds like a rebuild mandate, and it’s usually a documentation mandate wearing scarier clothes.
I tell anyone who inherits one of these environments to start writing restart runbooks the first week, long before anybody approves a rebuild. A rewrite takes years and gets cut in the next budget cycle. A runbook takes an afternoon per application, and it’s the thing the person on duty will reach for when the screens go dark.
My take
Demand a restart runbook for every application in the first week. Most software assumes it’ll never die, so after a crash you get unknown states and no procedure. Answer three things for each application, what state it can die in, how it gets back to a clean start, and what has to come up before and after it. That last answer becomes your restart sequence for the whole business. A rewrite takes years and gets cut. A runbook takes an afternoon.
What should executives ask about application recovery?
Your infrastructure people can likely tell you their recovery time for servers, storage, and networks, because the infrastructure world measures such things. Ask instead: if every application stopped mid-transaction right now, who knows the order and the procedures for bringing the business logic back, and where’s that written? In most companies the answer is a list of names, and names retire, take vacations, and, as my crash taught me twice in one day, occasionally hold the only copy of something the company can’t operate without.
The applications are the business, and I think it’s reckless to pour money into hardening the storage under them while nobody can say how the order system comes back after it dies. Every executive who signs off on that imbalance is betting the company on the one layer nobody ever tested, and the people who pay for the bet are the ones answering phones while the systems stay dark.
For more from this series, see The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
