Ask any vendor working on a problem what their current theory is, and treat the absence of one as the answer.
I learned to ask that question the hard way, during a week when the whole business crawled and nobody could tell me why. I’d rather hear a wrong theory on day one than a week of polite activity, because a wrong theory can be tested and thrown out. Silence burns time while everyone who depends on the system waits.
What causes a system to suddenly run slow?
Something changed. Monitoring shows software, load, and data volume, so that’s where everyone looks first.
Those three are visible. Dashboards report them and everyone knows how to talk about them.
I’d rather hear a wrong theory on day one than a week of polite activity, because a wrong theory can be tested and thrown out.
The floor underneath the software is invisible by comparison. Cables, ports, controllers, links, and the physical arrangement of equipment don’t appear on any dashboard, and they can change the behavior of everything above them.
A team can spend a very long time examining the visible layer while the answer sits in the layer nobody is instrumented to see. I think this blind spot is among the most expensive habits in IT. Teams trust whatever the dashboard shows them, and they ignore the physical layer until they’ve exhausted every other explanation and the business has already paid for the delay.
What does a disaster recovery mirror do?
Our production system copied its disks continuously to an identical system at a disaster recovery site about forty miles away. The copying ran asynchronously. That means production wrote its data, carried on working, and the copy caught up shortly afterward. Production never waited. The alternative is synchronous, where every write to production waits for confirmation that the remote site has it too. Synchronous gives you a perfect copy at every instant. It also means every single write pays the cost of the distance.
Over forty miles, that cost is significant. Asynchronous was the correct design and it worked for years.
How much can one bad cable slow down a business?
Ours slowed by at least a thousand percent.
Everything went slow at once. Operations that took minutes were taking hours. The business couldn’t function at that speed, and nothing about the software had changed. A slowdown like that is worse in some ways than an outage. When a system is down, everyone knows it and stops. When it crawls, people keep trying to work, the backlog grows by the hour, and the pressure on whoever runs the system gets heavier with every day nobody has an answer.
The cause was a bad cable. The bad cable had pushed the mirror from asynchronous to synchronous. Every write to production was now waiting for a confirmation from a site forty miles away, and the system was doing exactly what its configuration told it to do.
The software was fine. One failed piece of copper at the bottom of the stack had changed how the entire business behaved, and no amount of examining the software would have revealed it.
How long should a vendor take to find a problem?
A vendor should have a theory within a day, even a wrong one.
I called in a consulting firm first. They looked at it, and looked at it, and couldn’t work it out. A week went by with no answer and no hypothesis.
The cause was unusual and it lived in a layer most people don’t examine. They weren’t lazy and I have no complaint about their effort. But a week of effort with no theory is information. A team that knows a system produces a hypothesis early, tests it, and either narrows or discards it. A team without one produces activity, and activity looks like progress from the outside. Ask what the current theory is. Ask what has been ruled out. If neither question has an answer after several days, the issue isn’t effort. I don’t hold a grudge against the first firm. I do have a strong opinion about any vendor that keeps a client waiting for days without ever saying what it thinks is wrong. The client pays for that silence twice, once in the bill and again in a business that can’t run.
What makes an IT consultant worth keeping?
Somebody recommended a second firm, US Technical Services, so I called them.
I didn’t manage the conversation. I told them plainly that I had a disaster, that I was stuck, and that the firm I’d already hired wasn’t helping. No positioning, no defending the earlier decision.
They came in with their whole team. That was about five people at the time. They worked for a few hours. Then they pointed at a cable and said that’s the problem, switch it over. That was it. A week of one firm’s attention against a few hours of another’s, and the difference wasn’t effort or seniority. One team was searching and the other knew what to look at.
| Aspect | First firm | Second firm |
|---|---|---|
| Time on the problem | A week | A few hours |
| Theory | No answer and no hypothesis | Pointed at a cable and said switch it over |
| What they produced | Activity that looked like progress | The cause |
| Why | Searching | Knew what to look at |
Why would a vendor fix something for free?
I asked what it would cost. He said nothing. I asked what he meant, because the answer made no sense to me. He said that when your neighbor’s house is on fire, you come and put the fire out. You don’t charge them for it. You just put out the fire.
He became my consultant of choice permanently. That firm went on to run our private cloud, operate the computer room, take the overnight calls, and eventually rebuild the entire room to triple redundancy without taking anything down.
Years of work came out of a decision not to send an invoice for three hours.
I’m certain it was good business, and it didn’t read as a tactic at the time. That’s why it worked. A gesture calculated to earn future work announces itself. That one didn’t.
My take
Activity without a theory is a warning. If a vendor has spent days on your problem and can’t say what they think is wrong, bring in someone else. Our first firm spent a week with no hypothesis. The second found a bad cable in a few hours and declined to bill for it. That decision earned them years of work. Ask what’s been ruled out. Silence costs you twice, in the bill and in a business that can’t run.
How does a small company find good vendors?
Not through procurement. That’s the uncomfortable part.
Every company has a list of approved suppliers selected on price, references, and a scoring matrix.
That process produces adequate vendors, and it’s designed to. It was built to avoid bad outcomes, and nobody designed it to find exceptional ones. I think that’s a real weakness, because the crisis that matters most is the one where adequate won’t save you.
The ones that matter arrive another way. Somebody recommends them. They handle a crisis and you watch how they behave. They tell you something you didn’t want to hear. Two things help. Ask your network who they call when something has gone badly wrong, since that question gets a different answer than asking who they use. And pay attention to how a vendor behaves the first time something goes badly, because that’s the only real test and it only happens once.
I’ve never forgotten the man who told me you don’t charge your neighbor for putting out a fire. Vendors like that are rare, and I think companies that squeeze every supplier on price drive the good ones away and end up with firms that are careful, polite and useless when the building is burning.
Twenty years in the industry and nobody to call is how I found out I needed relationships. My book Networking Without the BS covers building the kind that hold when systems fail.
Frequently Asked Questions
- What is synchronous replication?, TechTarget
