A few months after I was hired to run the systems, I made what looked like the most boring request in IT. Our application logs were named with meaningless numbers, and hunting the right log meant guessing. I asked the programmers, a consulting firm writing the applications in a dialect of BASIC, to make the log names rational: include the program name, the date, something a human could read.
They did. They built the new name as program name, dot, date, dot, file type. Perfectly sensible. Except the operating system we ran permitted exactly one dot in a filename, name dot type, and the new format contained two. Every filename the change produced was illegal.
Three days in the dark
The morning after the change went live, the system didn’t come up. Not partially, not with errors we could chase. It simply didn’t come up, and here’s the exquisite part: there were no logs to tell us why, because the failure was the logging. Each program started, attempted to create its log file, was refused by the operating system, and stopped dead. The diagnostic trail and the disease were the same thing.
We spent three days down, three days awake, hunting a catastrophe. That’s what you hunt when the system is dead: something big, something structural. We were archaeologists looking for an earthquake, and the actual cause was a punctuation mark.
On the third day I got smart in the way you only get smart after exhaustion has burned off your assumptions. I gathered everybody and asked one question: tell me every single thing that changed. One person offered a library update, not it. Another offered something else, not it. And then someone mentioned, almost in passing, that the log filenames had changed, and described the format. Back out that change, I said. He did. We were up. That was the problem. That was it.
We spent three days hunting an earthquake. The cause was a punctuation mark.Share on X
How does a filename change cause a three-day outage?
Everything about that outage was out of proportion. A change too small to document took down the entire company for three days. The fix was one reversal; the search for it consumed seventy-plus hours of downtime and every ounce of the team’s credibility. And the question that solved it, what changed, cost nothing and could have been asked in the first ten minutes. Disasters don’t scale with the size of their causes.
Most of the worst outages I saw across twenty years of disaster recovery work traced back to something this size: a config line, a permission, a dot.
The operational lesson became doctrine for me in two parts. First, when a system that worked yesterday fails today, the first question isn’t “what’s broken,” it is “what changed,” asked of every person who could have changed anything, before any deeper archaeology begins. The answer is usually in the room and usually sounds too small to matter, so nobody volunteers it.
Second, changes get tested before they go live, all of them, with no change too trivial to qualify, because the filename change was precisely the kind that skips testing on the grounds of obvious harmlessness. I pushed for rigid test procedures after that, and got most of what I asked for; a funded staging environment eluded me. That taught its own lesson about how quickly organizations forget three-day outages once the systems hum again.
If you run a company, you don’t need to know what your teams changed last night. You need to know that someone can answer that question in five minutes when everything stops, because someday everything will stop, and the cause will be wearing its harmlessness as camouflage.
For more from this series, see The Disaster Recovery Hub: real disasters, real recoveries, and the plans that survive contact with reality.
