We Built a Disaster Recovery Site Without a Budget

I built a disaster recovery site with no budget, tested it for years, and never once ran the business on it. I’m proud of that site and uneasy about the gap, because testing the parts gave me evidence and only a live cutover would have given me proof. Most executives can’t tell those two apart, and they’re betting payroll, orders and customer records on the difference.

How do companies pay for work nobody approved?

By not asking for all of it at once.

Building a disaster recovery site was one of the main reasons I was hired, and it became my first major responsibility. There was no budget for it. I had money for machines. I had nothing for people, and I couldn’t hire anyone to do the work.

So I sent the people I already had out to the site periodically to build a piece of it, and I managed it that way. A trip here, a trip there, across years.

There was never a project. Never a headcount request, never an approval for the thing as a whole, never a line item anyone could point at and cut.

It existed because somebody kept sending people out there.

Is IT resilience usually built without a budget?

In companies under a couple hundred people, frequently yes, and it’s the opposite of how it gets sold.

The vendor version is a program with a start date, a budget, and a completion milestone. Somebody signs, work happens, and at the end you have a capability. The version that happens is a person who decides it matters and keeps chipping at it. No single trip needs approval, so no single trip gets refused.

That approach has an obvious weakness: it depends entirely on one person continuing to care. It also has one advantage nothing else offers. It survives budget cycles, because there’s nothing on the budget to cut. I don’t recommend it as a strategy. I recommend it as what you do when the alternative is having nothing, and I’d much prefer a company fund the work openly so it doesn’t die the week its champion leaves.

Why was the disaster recovery site only forty miles away?

Because that was the limit of the technology.

The site sat about forty miles from the main computer room. That distance was as far as we could go and still keep the systems copying properly.

Connectivity started as a few circuits, became a faster one, and eventually became something much faster than that. Every improvement changed what was possible, and bandwidth was the binding constraint on how good the copy could be.

What’s worth noticing is what a hard constraint does to a decision. Forty miles was argued about, measured, and understood by everyone involved, because we had no choice about it. Today distance is nearly free. A company can put a copy of its data anywhere, and because the tradeoff disappeared, almost nobody reasons about it. When a constraint is forced, people think about it. When it vanishes, they stop. That’s backwards. Cheap distance should have made recovery better everywhere, and I think it’s made a lot of people lazy about it: the copy lands in whatever region was the default, and nobody can say whether one storm or one provider outage takes out both.

What does a disaster recovery site need besides computers?

Somewhere for people to sit and work.

Ours had a duplicate machine running, receiving data, building logs. That part gets all the attention and it’s the part everyone plans.

It also had laptops, computers, and a full twenty-desk setup, ready to go.

That was deliberate. A disaster that takes out a computer room usually takes out the building the computer room is in, and the people who run the business need somewhere to be. A recovery plan that restores the systems and leaves forty people with nowhere to sit has solved half the problem.

What is the difference between a tested system and a proven one?

We tested that site. We never failed over to it live.

The testing was real. Systems were exercised, the copy was verified, the equipment was checked. That work wasn’t theater and it caught things. What never happened was the business running on it. No day where the main site was gone and the company operated from forty miles away.

My word for that is fortunately.

Testing establishes that the parts work under known conditions. A live cutover establishes that the system handles real load, real users, and real decisions made under pressure by people who have never done it before. Only the cutover counts as proof, and almost nobody has it. That gap worries me more than any missing piece of hardware. Until a plan has carried the business, it’s a hypothesis, and the people who find out whether it holds are the staff who show up to a building with dark screens and the customers waiting on them.

Should you run a full failover test?

Most companies should and most won’t, and the reasons are legitimate. A real failover test means taking the business down deliberately. It means accepting that something might not come back, and it means somebody has to authorize a self-inflicted outage on a normal Tuesday. The version that works for a smaller company is narrower. Fail over one system and not everything. Do it during a quiet period. Have the people who would run the recovery do it, instead of the person who designed it.

That last detail outweighs the scale. A recovery executed by its designer proves the design. A recovery executed by whoever is on duty proves the documentation, and the documentation is what you’ll have. I’d tell any IT director to hand the runbook to the newest person on the night shift and watch. If they can’t bring one system back from the documentation alone, you’ve learned something on a quiet evening that you’d otherwise learn in the middle of a real disaster.

What should you tell your board about recovery readiness?

The truth about which kind of evidence you hold. We have a recovery site and we test it regularly is accurate and honest. We’re covered implies something stronger, and the gap between those two statements is where companies discover their plan was a belief.

Saying it plainly costs nothing and it changes the conversation. A board told that the parts are tested and the whole has never been exercised will usually ask what a full test would take. That’s exactly the question you want them to ask.

Every company with a backup strategy sits somewhere on that line, and leaders who tell their boards “we’re covered” when all they’ve got is a tested site are fooling themselves. That sentence moves the risk off their desk and onto every employee and customer who’d depend on the plan working. Say which kind of evidence you hold, and then go get the better kind.

Frequently Asked Questions