A few months after I was hired to run the systems, I made what looked like the most boring request in IT. Our application logs were named with meaningless numbers, and hunting the right log meant guessing. I asked the programmers, a consulting firm writing the applications in a dialect of BASIC, to make the log names rational: include the program name, the date, something a human could read.
They did. They built the new name as program name, dot, date, dot, file type. Perfectly sensible. Except the operating system we ran permitted exactly one dot in a filename, name dot type, and the new format contained two. Every filename the change produced was illegal.
I’ve never let myself forget that I asked for that change. The programmers built exactly what I requested, and nobody in the room, me included, stopped to ask what the operating system would do with it. That part still stings.
Three days in the dark
The morning after the change went live, the system didn’t come up. Not partially, not with errors we could chase. It simply didn’t come up, and the exquisite part was that there were no logs to tell us why, because the failure was the logging. Each program started and attempted to create its log file. The operating system refused, and the program stopped dead. The diagnostic trail and the disease were the same thing.
We spent three days down, three days awake, hunting a catastrophe. That’s what you hunt when the system is dead: something big, something structural. We were archaeologists looking for an earthquake, and the actual cause was a punctuation mark.
The diagnostic trail and the disease were the same thing.
Operations teams reward whoever finds the deep, clever cause, and mine was no different. So everyone went hunting for one, and the boring question sat unasked while the business stayed dark for another day. I think that instinct costs companies more downtime than any hardware failure does.
On the third day I got smart in the way you only get smart after exhaustion has burned off your assumptions.
I gathered everybody and asked one question: tell me every single thing that changed. One person offered a library update, not it. Another offered something else, not it. And then someone mentioned, almost in passing, that the log filenames had changed, and described the format. Back out that change, I said. He did. We were up. That was the problem. That was it.
My take
Ask what changed, first, every time. A system dies, everyone goes hunting for something big and clever, and the boring question sits unasked. Our three-day outage came from a log filename with one dot too many, a change I’d asked for myself. One command fixed it. Make every person in the room list every change, however trivial. Small edits like that cause more outages than the dramatic failures people go looking for.
We spent three days hunting an earthquake. The cause was a punctuation mark.Share on X
How does a filename change cause a three-day outage?
Everything about that outage was out of proportion.
A change too small to document took down the entire company for three days. The fix was one reversal, and the search for it consumed seventy-plus hours of downtime and every ounce of the team’s credibility. The question that solved it, what changed, cost nothing and could have been asked in the first ten minutes. I’m still irritated about those ten minutes.
Most of the worst outages I saw across twenty years of disaster recovery work traced back to something this size: a config line, a permission, a dot.
The operational lesson became doctrine for me in two parts. First, when a system that worked yesterday fails today, the first question isn’t “what’s broken,” it is “what changed,” asked of every person who could have changed anything, before any deeper archaeology begins. The answer is usually in the room and usually sounds too small to matter, so nobody volunteers it.
Second, changes get tested before they go live, all of them, with no change too trivial to qualify, because the filename change was exactly the kind that skips testing on the grounds of obvious harmlessness. I pushed for rigid test procedures after that, and got most of what I asked for; a funded staging environment eluded me. That taught its own lesson about how quickly organizations forget three-day outages once the systems hum again.
If you run a company, you need someone who can tell you what changed last night within five minutes of everything stopping. The phrase “it’s just a small change” is a warning sign. Every team that uses it to skip testing is betting the whole company on a guess, and the people who pay are the ones awake at the console and the customers who can’t be served.
For more from this series, see The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
