Ask any vendor working on a problem what their current theory is, and treat the absence of one as the answer.
What causes a system to suddenly run slow?
Something changed. Monitoring shows software, load, and data volume, so that is where everyone looks first.
Those three are visible. Dashboards report them and everyone knows how to talk about them.
The floor underneath the software is invisible by comparison. Cables, ports, controllers, links, and the physical arrangement of equipment do not appear on any dashboard, and they can change the behavior of everything above them.
A team can spend a very long time examining the visible layer while the answer sits in the layer nobody is instrumented to see.
What does a disaster recovery mirror do?
Our production system copied its disks continuously to an identical system at a disaster recovery site about forty miles away.
The copying ran asynchronously, which means production wrote its data, carried on working, and the copy caught up shortly afterward. Production never waited.
The alternative is synchronous, where every write to production waits for confirmation that the remote site has it too. Synchronous gives you a perfect copy at every instant. It also means every single write pays the cost of the distance.
Over forty miles, that cost is significant. Asynchronous was the correct design and it worked for years.
How much can one bad cable slow down a business?
Ours slowed by at least a thousand percent.
Everything went slow at once. Operations that took minutes were taking hours. The business could not function at that speed, and nothing about the software had changed.
The cause was a bad cable.
The bad cable had pushed the mirror from asynchronous to synchronous. Every write to production was now waiting for a confirmation from a site forty miles away, and the system was doing exactly what its configuration told it to do.
The software was fine. One failed piece of copper at the bottom of the stack had changed how the entire business behaved, and no amount of examining the software would have revealed it.
How long should a vendor take to find a problem?
A vendor should have a theory within a day, even a wrong one.
I called in a consulting firm first. They looked at it, and looked at it, and could not work it out. A week went by with no answer and no hypothesis.
I want to be fair to them. The cause was unusual and it lived in a layer most people do not examine. They were not lazy and I have no complaint about their effort.
But a week of effort with no theory is information. A team that knows a system produces a hypothesis early, tests it, and either narrows or discards it. A team without one produces activity, and activity looks like progress from the outside.
Ask what the current theory is. Ask what has been ruled out. If neither question has an answer after several days, the issue is not effort.
What makes a consultant worth keeping?
Somebody recommended a second firm, US Technical Services, so I called them.
I did not manage the conversation. I told them plainly that I had a disaster, that I was stuck, and that the firm I had already hired was not helping. No positioning, no defending the earlier decision.
They came in with their whole team, which was about five people at the time. They worked for a few hours. Then they pointed at a cable and said that is the problem, switch it over.
That was it. A week of one firm’s attention against a few hours of another’s, and the difference was not effort or seniority. One team was searching and the other knew what to look at.
Why would a vendor fix something for free?
I asked what it would cost. He said nothing.
I asked what he meant, because the answer made no sense to me. He said that when your neighbor’s house is on fire, you come and put the fire out. You do not charge them for it. You just put out the fire.
He became my consultant of choice permanently. That firm went on to run our private cloud, operate the computer room, take the overnight calls, and eventually rebuild the entire room to triple redundancy without taking anything down.
Years of work came out of a decision not to send an invoice for three hours.
I am certain it was good business, and it did not read as a tactic at the time, which is why it worked. A gesture calculated to earn future work announces itself. That one did not.
How does a small company find vendors like that?
Not through procurement, which is the uncomfortable part.
Every company has a list of approved suppliers selected on price, references, and a scoring matrix. That process produces adequate vendors and it is designed to, because it is built to avoid bad outcomes rather than to find exceptional ones.
The ones that matter arrive another way. Somebody recommends them. They handle a crisis and you watch how they behave. They tell you something you did not want to hear.
Two things help. Ask your network who they call when something has gone badly wrong, since that question gets a different answer than asking who they use. And pay attention to how a vendor behaves the first time something goes badly, because that is the only real test and it only happens once.
The Guides That Get Your Book Written, Published, and Sold
Four short, practical guides on writing, publishing, and selling your book, plus the occasional note when there's something worth your time. No fluff, no daily inbox clutter. Drop your email and they're yours.
We use MailerLite to manage our list and send these emails. Your address is used only to send you what you signed up for. We will not sell it, share it, or use it for anything else, and you can unsubscribe anytime.