Write down which constraints are fixed before anyone starts proposing solutions.
This story is great because it cuts against the way most teams treat a constraint. They fight it, complain about it, and try to negotiate it away. I’d rather have the hard limit handed to me on day one, because the limit is what forces people to stop reciting the standard procedure and start thinking about the actual problem.
Why do constraints produce better solutions?
Because a team with room to maneuver picks the obvious answer, and the obvious answer is obvious to everybody.
Give a group unlimited time and budget and they’ll schedule the outage, move the systems, do the work, and move them back. That’s a fine plan and it requires no thought.
Remove the outage from the table and something different happens. The obvious answer is gone, so the team has to look at the actual problem and not at the standard procedure for problems of that shape. Most good engineering I’ve watched came out of somebody being told they couldn’t do the normal thing.
What happens when a data center floor starts to fail?
The consultant running our computer room came to me one day with a problem. A raised floor tile under the main cabinet was buckling. Buckling is worse than cracking or wear, because it meant the tile could give way at any moment.
Most good engineering I’ve watched came out of somebody being told they couldn’t do the normal thing.
Above that tile sat the cabinet holding eight machines running the business. This had to be fixed immediately, so we took the next weekend. A week to plan, a weekend to execute.
How do you replace a floor tile under live equipment?
The obvious approach failed before we started. Move the workloads elsewhere, take the cabinet down, replace the tile, bring it back. That’s what anyone would propose and it’s what I would have proposed.
We didn’t have the spare capacity to relocate eight machines. We moved a few and couldn’t move the rest. Then my boss added the constraint that settled everything. Zero downtime. The business wouldn’t stop for this.
Now the standard answer was unavailable, and so was the fallback. What remained was one option nobody would pick from a menu.
Lift the cabinet while it was live and running. Replace the tile underneath it. Set it back down.
How do you lift a running computer off the floor?
I walked into the computer room and there was an enormous inflatable bladder under the cabinet, with bracing everywhere.
It looked like a nightmare. It was also clearly not going to fall over. That’s the impression good rigging gives you. They lifted it. They replaced the tile. They set it back down. It went very well, and it was expensive.
Nothing about the operation was improvised. The firm was US Technical Services, and the people doing the work were ex-Marines and ex-special forces. They had clipboards. They had a written plan and they followed it exactly, and I’ve rarely respected a crew more.
A dangerous operation was boring to watch. That’s what competence looks like from the outside.
Why does written procedure matter more than skill?
Skill fails under pressure and procedure doesn’t.
The people lifting that cabinet were highly capable, and that’s not why it worked. It worked because somebody had written down the sequence, everyone knew their part, and nobody had to make a judgment call while several tons of equipment sat on an air bladder. The failure mode in this kind of work is the risky operation performed by people improvising, where two competent people each make a reasonable decision that contradicts the other.
Most companies have never written down what happens during their equivalent moment. They find out how well they improvise while the thing is already buckling. I think that’s negligence dressed up as confidence, and the people who pay for it are the staff standing in the room when it goes wrong.
What should a small company write down before it needs to?
The three or four moments where being wrong is expensive.
A page each, no binder, for the things that would hurt. Restoring from backup. Failing over to a second site. Bringing the business up after an extended outage. Handling the discovery that something physical is about to fail. Each page says who does what, in what order, and who decides. Nothing else goes in, and writing one takes an afternoon. Writing it forces the decisions in advance, when there’s time to think, instead of at the moment when everyone is standing around looking at a problem. The value is in those decisions, and the paper only proves somebody made them.
My take
Hard constraints hand you answers you’d never pick off a menu. Zero downtime meant we lifted a live cabinet off the floor to replace a tile, and it went very well. Skill didn’t make it work. A written plan followed exactly did. Write your procedures down before you need them, especially for anything dangerous or rare. A crew with clipboards beats a crew of geniuses improvising.
How do you decide what your boss should see?
My boss wanted to watch the lift. I told him no.
He asked why, and I told him the truth: watching it would probably upset him. He accepted that and stayed out of the computer room.
I didn’t hide the operation from him. He knew what we were doing, he’d set the constraint that shaped it, and he’d approved the cost. What I declined was the viewing.
There’s a version of managing upward that means controlling what the person above you knows, and that version ends badly. This was different. He had every fact, and I made a judgment about what would help him and what would only make him anxious about a decision already made.
He trusted the judgment. That’s worth more than any report, and it only exists because of everything that came before it. I think a boss who insists on watching every risky operation hasn’t built that trust with his people yet, and no dashboard will build it for him. It takes years of being told the truth and seeing it hold up.
