Ask any vendor working on a problem what their current theory is, and treat the absence of one as the answer.
What causes a system to suddenly run slow?
Something changed. Monitoring shows software, load, and data volume, so that’s where everyone looks first.
Those three are visible. Dashboards report them and everyone knows how to talk about them.
The floor underneath the software is invisible by comparison. Cables, ports, controllers, links, and the physical arrangement of equipment don’t appear on any dashboard, and they can change the behavior of everything above them.
A team can spend a very long time examining the visible layer while the answer sits in the layer nobody is instrumented to see.
What does a disaster recovery mirror do?
Our production system copied its disks continuously to an identical system at a disaster recovery site about forty miles away.
The copying ran asynchronously. That means production wrote its data, carried on working, and the copy caught up shortly afterward. Production never waited.
The alternative is synchronous, where every write to production waits for confirmation that the remote site has it too. Synchronous gives you a perfect copy at every instant. It also means every single write pays the cost of the distance.
Over forty miles, that cost is significant. Asynchronous was the correct design and it worked for years.
How much can one bad cable slow down a business?
Ours slowed by at least a thousand percent.
Everything went slow at once. Operations that took minutes were taking hours. The business couldn’t function at that speed, and nothing about the software had changed.
The cause was a bad cable.
The bad cable had pushed the mirror from asynchronous to synchronous. Every write to production was now waiting for a confirmation from a site forty miles away, and the system was doing exactly what its configuration told it to do.
The software was fine. One failed piece of copper at the bottom of the stack had changed how the entire business behaved, and no amount of examining the software would have revealed it.
How long should a vendor take to find a problem?
A vendor should have a theory within a day, even a wrong one.
I called in a consulting firm first. They looked at it, and looked at it, and couldn’t work it out. A week went by with no answer and no hypothesis.
I want to be fair to them. The cause was unusual and it lived in a layer most people don’t examine. They weren’t lazy and I have no complaint about their effort.
But a week of effort with no theory is information. A team that knows a system produces a hypothesis early, tests it, and either narrows or discards it. A team without one produces activity, and activity looks like progress from the outside.
Ask what the current theory is. Ask what has been ruled out. If neither question has an answer after several days, the issue isn’t effort.
What makes an IT consultant worth keeping?
Somebody recommended a second firm, US Technical Services, so I called them.
I didn’t manage the conversation. I told them plainly that I’d a disaster, that I was stuck, and that the firm I’d already hired wasn’t helping. No positioning, no defending the earlier decision.
They came in with their whole team. That was about five people at the time. They worked for a few hours. Then they pointed at a cable and said that’s the problem, switch it over.
That was it. A week of one firm’s attention against a few hours of another’s, and the difference wasn’t effort or seniority. One team was searching and the other knew what to look at.
Why would a vendor fix something for free?
I asked what it would cost. He said nothing.
I asked what he meant, because the answer made no sense to me. He said that when your neighbor’s house is on fire, you come and put the fire out. You don’t charge them for it. You just put out the fire.
He became my consultant of choice permanently. That firm went on to run our private cloud, operate the computer room, take the overnight calls, and eventually rebuild the entire room to triple redundancy without taking anything down.
Years of work came out of a decision not to send an invoice for three hours.
I’m certain it was good business, and it didn’t read as a tactic at the time. That’s why it worked. A gesture calculated to earn future work announces itself. That one didn’t.
How does a small company find good vendors?
Not through procurement, which is the uncomfortable part.
Every company has a list of approved suppliers selected on price, references, and a scoring matrix. That process produces adequate vendors and it’s designed to, because it’s built to avoid bad outcomes and not to find exceptional ones.
The ones that matter arrive another way. Somebody recommends them. They handle a crisis and you watch how they behave. They tell you something you didn’t want to hear.
Two things help. Ask your network who they call when something has gone badly wrong, since that question gets a different answer than asking who they use. And pay attention to how a vendor behaves the first time something goes badly, because that’s the only real test and it only happens once.