Write down which constraints are fixed before anyone starts proposing solutions.
Why do constraints produce better solutions?
Because a team with room to maneuver picks the obvious answer, and the obvious answer is obvious to everybody.
Give a group unlimited time and budget and they’ll schedule the outage, move the systems, do the work, and move them back. That’s a fine plan and it requires no thought.
Remove the outage from the table and something different happens. The obvious answer is gone, so the team has to look at the actual problem and not at the standard procedure for problems of that shape.
Most good engineering I’ve watched came out of somebody being told they couldn’t do the normal thing.
What happens when a data center floor starts to fail?
The consultant running our computer room came to me one day with a problem.
A raised floor tile under the main cabinet was buckling. Not cracked, not worn. Buckling, which meant it could give way at any moment.
Above that tile sat the cabinet holding eight machines running the business.
This had to be fixed immediately, so we took the next weekend. A week to plan, a weekend to execute.
How do you replace a floor tile under live equipment?
The obvious approach failed before we started.
Move the workloads elsewhere, take the cabinet down, replace the tile, bring it back. That’s what anyone would propose and it’s what I would have proposed.
We didn’t have the spare capacity to relocate eight machines. We moved a few and couldn’t move the rest.
Then my boss added the constraint that settled everything. Zero downtime. The business wouldn’t stop for this.
Now the standard answer was unavailable, and so was the fallback. What remained was one option nobody would pick from a menu.
Lift the cabinet while it was live and running. Replace the tile underneath it. Set it back down.
How do you lift a running computer off the floor?
I walked into the computer room and there was an enormous inflatable bladder under the cabinet, with bracing everywhere.
It looked like a nightmare. It was also clearly not going to fall over, which is the impression good rigging gives you.
They lifted it. They replaced the tile. They set it back down. It went very well, and it was expensive.
Nothing about the operation was improvised. The firm was US Technical Services, and the people doing the work were ex-Marines and ex-special forces. They’d clipboards. They’d a written plan and they followed it exactly.
A dangerous operation was boring to watch. That’s what competence looks like from the outside.
Why does written procedure matter more than skill?
Skill fails quietly under pressure and procedure doesn’t.
The people lifting that cabinet were highly capable, and that’s not why it worked. It worked because somebody had written down the sequence, everyone knew their part, and nobody had to make a judgment call while several tons of equipment sat on an air bladder.
The failure mode in this kind of work is never the risky operation. It’s the risky operation performed by people improvising, where two competent people each make a reasonable decision that contradicts the other.
Most companies have never written down what happens during their equivalent moment. They find out how well they improvise while the thing is already buckling.
What should a small company write down before it needs to?
The three or four moments where being wrong is expensive.
Not a binder. A page each for the things that would hurt. Restoring from backup. Failing over to a second site. Bringing the business up after an extended outage. Handling the discovery that something physical is about to fail.
Each page says who does what, in what order, and who decides. Nothing else goes in, and writing one takes an afternoon.
The value isn’t the paper. The value is that writing it forces the decisions in advance, when there’s time to think, and not at the moment when everyone is standing around looking at a problem.
How do you decide what your boss should see?
My boss wanted to watch the lift. I told him no.
He asked why, and I told him the truth, which was that watching it would probably upset him. He accepted that and stayed out of the computer room.
I didn’t hide the operation from him. He knew what we were doing, he’d set the constraint that shaped it, and he’d approved the cost. What I declined was the viewing.
There’s a version of managing upward that means controlling what the person above you knows, and that version ends badly. This was different. He’d every fact and I made a judgment about what would help him and what would only make him anxious about a decision already made.
He trusted the judgment. That’s worth more than any report, and it only exists because of everything that came before it.