They Hired Us to Settle an Argument, Not to Answer a Question

Ask who’s going to use the report and what they want it to say, before you agree to write it.

Honest work can still serve a loaded question, and I think more technical people should admit how often that happens.

Nobody lies to you. The client pays for real engineering and gets it. But the report was always going to be ammunition in somebody else’s fight, and the people who pay for the fight are usually the ones waiting on the outcome, stuck with equipment they can’t use while the two sides trade positions.

Why do companies commission technical assessments?

Three reasons, and only one of them is the reason people give. The stated reason is to learn something nobody knows. Sometimes that’s true, and those are good projects to work on.

The second reason is to create a record. Somebody needs a document showing that a competent outside party looked at the situation, usually because a decision has already been made and needs support.

The third reason is to win an argument with a counterparty. Two sides disagree about money, and an assessment becomes evidence one of them intends to use.

All three produce real work for the people doing the assessment, though only the first reason starts with an open question.

The second and third kinds are fine by me as long as everybody says out loud which one they’re buying. What I resent is the pretense. When everyone in the room calls it fact-finding while each side already knows the answer it wants, the engineers become cover and the budget pays for theater.

What happens when a building burns and the computers are in the basement?

On the night of October 15, 1984, a four-alarm fire broke out at 475 L’Enfant Plaza in Washington DC, the building that housed both the US Postal Service headquarters and PBS. It started on the ninth floor shortly after eleven and burned through the upper floors. Around 190 firefighters and 48 pieces of equipment brought it under control in about an hour.

Water goes down. It went through the building and into the computer room in the basement, and the machines were flooded.

PBS put the preliminary damage to its property at more than twelve million dollars. A spokeswoman listed broadcasting transmission equipment, tape and editing machines, and computers. Insurance investigators toured the building the next day. An insurance company hired one of the Big Eight firms, and that firm hired us. The question they put to us was whether the computers still worked and, more to the point, whether they were still accurate. Nobody had done this before as far as anyone knew. There was no procedure and no prior art. We designed programs from scratch that ran the equipment and compared the output against known good results. Most of it was written in the general-purpose language of the era. I wrote my portion in assembler because I was fast in it and the work needed to be quick.

How do you know if a computer still gives the right answer?

You give it work whose answer you already know, and you check. The hard part is the reference. Something has to establish what the right answer looks like, and the machines that would normally tell you are the ones in question.

The deeper problem is that damaged equipment doesn’t fail cleanly. A machine that won’t turn on is easy. A machine that turns on, runs, produces output, and is wrong once every few thousand operations is the situation that ruins a company. Every restore from backup after a ransomware incident faces the same question and almost nobody asks it. The system came back. Nobody checked whether the data came back correct. Working and correct are separate conditions. Confirming the first is easy and most organizations stop there.

I think this is among the most dangerous gaps in IT today. Companies celebrate the restore, send the all-clear, and months later discover the books have been wrong since the incident. The people who pay are the customers billed from bad data and the staff blamed for mistakes the system made.

What were they really arguing about?

Money, and the gap was smaller than anyone would guess. The insurance company wanted to pay a certain amount. PBS wanted new computers. Our project existed to establish whether refurbishing the flooded equipment would cost more than replacing it, and to establish it straight.

The difference between the two positions was around twenty percent. That’s a disagreement, not a chasm. Nobody was deceived about any of this. The client told us plainly what the project was for. We knew from the beginning and we did honest work anyway. The engineering was real. The findings were real. The project was still an instrument in a negotiation between two parties who each wanted a particular answer.

Why do assessments take longer than the decision is worth?

Because when two sides are using the assessment to argue, neither one is in a hurry for it to end.

The work continued for a long time. We never really finished it. I left the project and never learned the verdict.

From my seat it was a good assignment. There was real money to spend, I stayed in the best hotels in Washington, and I visited the Smithsonian. It remains the only time I’ve been to Washington DC and I enjoyed it.

Meanwhile the flooded computers sat there and the argument kept running. That part still rankles. Both sides meant well, and every month of the dispute was another month of equipment nobody trusted sitting in a basement while people like me got paid to study it. An assessment commissioned to answer a question ends when the question is answered. An assessment commissioned to support a position ends when somebody decides to stop, and that decision has nothing to do with the findings.

What did the delay end up costing?

More than the amount in dispute.

The insurance company eventually said buy the new computers. They paid considerably more than the original settlement would have cost them, after months of assessment work on top.

Both sides were behaving rationally the entire time. The insurer wanted to avoid overpaying. PBS wanted equipment they could trust. Neither position was unreasonable and the combined outcome was worse for everyone than an early agreement would have been.

I put the fault on the arrangement more than on either party. Nobody in it had a reason to call time, and nobody was rewarded for proposing the cheap, obvious settlement on day one.

That pattern turns up in vendor disputes, contract renegotiations, and build-versus-buy analyses at every size of company. Two reasonable positions, an expensive process to decide between them, and a total cost exceeding the gap.

What should you ask before commissioning an assessment?

Four questions, and the answers change how you run the project.

What decision does this report feed into? Who else is going to read it, and do they have a position already? What happens if the finding goes the other way? And what’s the difference between the two positions, in money?

The last question is the one people skip. If an assessment costs a meaningful fraction of the amount being argued over, the assessment isn’t a way to save money. Splitting the difference on day one is frequently the cheaper answer and nobody proposes it, because proposing it feels like conceding. None of this argues against assessments. Plenty of them earn their cost several times over. It argues for knowing which of the three kinds you’re buying before you sign.

I’d tell anyone about to commission one of these to say plainly to themselves what it’s for. If the report is ammunition, call it ammunition, cap what you’ll spend on it, and put a date on the calendar when you’ll settle regardless. Letting an assessment run open-ended because neither side wants to blink is how a twenty percent disagreement turns into a bill bigger than the disagreement ever was.

From the author

Build a Profitable Consulting Practice
Engagements like that one are what Build a Profitable Consulting Practice is about. Expertise does not pay the bills, contracts do, and the book covers forty years of learning how.

Frequently Asked Questions