We Did Not Fix the System, We Moved It

This entry is part 14 of 21 in the series The Operations Room
TL;DR: Before you fund a project to modernize an aging system, find out whether the failures are coming from the software or from the hardware underneath it. We had an old system breaking constantly and running slow. Nobody rewrote a line of it. We put it on new hardware through virtualization and the problem went away for six years.

Check whether the failures track the software or the machine, because that one question decides how much the fix costs.

Why do old systems break more often over time?

Two separate reasons that get treated as one.

Software decays in a specific sense. Dependencies age, assumptions stop holding, and the people who understood it move on. That decay is real and rewriting is the only cure.

Hardware decays in a physical sense. Components fail, capacity that was generous becomes tight, and a machine that ran a workload comfortably in year three struggles in year nine.

Both produce the same symptom, which is a system that breaks more than it used to. Almost every modernization proposal assumes the first cause without checking for the second.

How do you tell software problems from hardware problems?

Look at what the failures have in common.

Software failures cluster around behavior. A specific function fails, a particular data condition breaks something, the same error appears under the same circumstances. There is a pattern you can describe.

Hardware failures cluster around load and time. Things get slow when the machine is busy. Failures happen at no consistent place in the code. Restarting helps for a while.

The second pattern points at the floor rather than the building. When somebody describes a system as breaking all the time and being slow, without being able to say what it breaks on, that is usually the machine.

What happens when you move an old system to new hardware?

We had a large business system on old hardware. It broke constantly and it was slow.

Upgrading the software itself was not on the table. The system was old enough that we believed touching it would break it, and that belief was reasonable given what we knew.

Working with a consultant, I proposed a different approach. Virtualize it. Put the same software, unchanged, onto a virtual machine running on modern hardware.

The theory was that the software would run faster and stop failing, because the failures were coming from underneath it rather than from inside it.

We moved it, and it ran like a bat out of hell. It stopped breaking, because the hardware under it was now stable. It stopped having issues. And it stayed there, working, for about six years.

Nobody wrote a line of code. Nobody paid down a line of technical debt. The problem went away because the floor got replaced.

Why should you buy more capacity than you need?

Because getting approval for a large purchase is no harder than getting approval for a small one.

When I bought the host for that project, I bought a big one. The single system did not require it. I bought it because I intended to virtualize far more if the first move worked, and because I had learned something years earlier about how approvals work.

The effort of an approval is roughly fixed. The same meetings, the same justification, the same finance conversation, whether the number is small or large. What changes is the number, not the work.

And if it gets cut, a request sized with room to spare gets cut back to something still workable. A request sized exactly to the need gets cut into the bone.

What does spare capacity buy you later?

Spare capacity buys the project you could not have proposed at the time.

That oversized host is why one rescued system turned into a virtualized computer room. Nobody approved that outcome, because nobody could have described it at the time. It became possible because the first purchase was not sized to the first problem.

A request scoped to one need is honest, defensible, and leaves you with exactly one solved problem. That is sometimes the right answer, particularly when money is tight and nobody trusts the plan yet.

It is also how a company ends up buying five things across five years, each with its own approval cycle, when one purchase would have covered all of them.

Should a small company still run its own hardware?

Sometimes, and the answer has shifted twice in twenty years.

When I built that private cloud, running your own was the only real option for a workload like ours. Then the market moved and everything went to hosted infrastructure. Now a significant share of companies are moving workloads back.

The reason for the return is unglamorous. Steady-state workloads that do not vary much never needed elastic capacity, and they look expensive once somebody measures them against owning the equivalent.

The test is variability. A workload that spikes unpredictably belongs somewhere elastic. A workload that runs at roughly the same level all year is a candidate for hardware you own, and that describes a lot of what a forty-person company runs.

When is moving a system better than rewriting it?

When the software still does its job and the environment underneath it is the problem.

That condition is narrower than most modernization proposals assume. Plenty of old systems do need rewriting, because the business has changed, the rules are trapped in code nobody can read, or nobody left understands it.

But the diagnosis gets skipped constantly. A system breaks, somebody proposes replacing it, and the proposal is approved without anyone establishing what was breaking or why.

Six years of stability came out of moving something instead of rebuilding it. The rebuild would have cost more, taken longer, and carried the risk of losing behavior nobody had documented.

Ask what is breaking before you decide what to build.

The Guides That Get Your Book Written, Published, and Sold

Four short, practical guides on writing, publishing, and selling your book, plus the occasional note when there's something worth your time. No fluff, no daily inbox clutter. Drop your email and they're yours.

We use MailerLite to manage our list and send these emails. Your address is used only to send you what you signed up for. We will not sell it, share it, or use it for anything else, and you can unsubscribe anytime.

Frequently Asked Questions

What is virtualization in plain terms?
Software that lets one physical machine run several separate systems, each behaving as though it has its own computer. The systems do not know they are sharing, which is why old software can be moved onto new hardware without being changed.
Can you virtualize any old system?
Most of them, with exceptions for software tied to specific physical hardware or licensed against a particular machine. Test with a copy before committing, since the failures that do occur tend to appear immediately rather than later.
How do you justify buying more capacity than you need?
Name what the spare capacity is for, even roughly. Unexplained headroom reads as padding and gets cut. Headroom described as the next three systems you intend to move reads as a plan.
Is a private cloud worth it for a small company?
It can be, for steady workloads that do not vary much through the year. The comparison is total cost against hosted infrastructure over three to five years, including the staff time to run it, which is the part usually left out.
How long should a virtualized legacy system be expected to last?
Longer than most people plan for. Ours ran six years. The end usually arrives from outside, when a vendor stops supporting something or a business requirement changes, rather than from the system itself degrading.
Does moving a system instead of rewriting it just delay the problem?
If the underlying issue is the software, yes, and the delay costs money. If the issue is the hardware, moving it solves the problem outright. Establishing which one you have is the difference between a cheap fix and a postponed bill.

📁︎ Technology

🏷︎ Infrastructure🏷︎ IT Operations🏷︎ Technology

📝 Disclaimer

The views and opinions expressed in this blog post are solely those of Richard Lowe and are based on personal experience and research. This content is for informational purposes only and should not be construed as professional legal, financial, accounting, or business advice. Always consult with qualified professionals before making important business or legal decisions. Richard Lowe is not a lawyer, accountant, or licensed professional advisor, and this content does not establish any professional relationship.

0 comments

No comments yet. Yours can be the first.

Was this useful?

Leave a comment