When the performance crisis hit after our big-bang platform migration, the emergency fix bought survival, and survival bought time to find the cause.
The cause was the query layer. Hundreds of SQL queries, written and tuned over years against the old database, behaved badly against the new one. Each query was logically correct. The data came back right. It just came back slowly, and hundreds of slow queries compound into a platform that can’t carry a business.
The query-layer problem still rankles. Nobody lied to us, and nobody made a stupid mistake. The cost was simply invisible until the whole company was standing on the new platform, and by then the only question left was how fast we could pay it. Every executive approving a database migration deserves to hear about it before anyone tells them everything will carry over.
Why do correct queries go bad after migration?
Migration budgets detonate on a mechanism executives can understand without jargon. A database is an engine that decides how to find what you ask for, and every database engine decides differently. A query shaped to be fast on one engine can be pathologically slow on another. Nothing is broken. The new engine just takes a different path to the same answer, and nobody selling the new platform is in a hurry to mention it.
Multiply that by every query your business has accumulated over decades, and the migration has silently converted your best-tuned asset into your largest liability.
Nothing in our testing fully exposed this, because the pile-up only appears under real production load, all the queries, all the users, all at once. Which is why the bill arrived after go-live, in the barely-tolerable weeks, instead of in any planning document.
Why did testing miss the migration problem?
A query shaped to be fast on one engine can be pathologically slow on another.
The fair question is why none of this came out before go-live, and the answer is a lesson about what testing can and can’t see. Individual queries tested fine, because a query that takes four times longer on the new engine still returns in what looks like acceptable time when it runs alone against a test copy. The pathology was compound: hundreds of modestly degraded queries, sharing one platform, under full production concurrency, with real data distributions the test sets only approximated.
Each ingredient was individually testable; the collision of all of them existed only in production.
Load testing as a discipline barely existed for us then. Today it exists, teams still skip it to hit a date, and I don’t understand how anyone signs off on that. The date gets met, and then the business spends months paying for the shortcut.
If you want the pre-payment version of our lesson: inventory the query workload before committing to the migration, benchmark a representative slice, the heaviest and most frequent queries, against the target platform with production-scale data, and multiply what you find across the full inventory. The number that exercise produces is the real migration estimate. Ours arrived by invoice instead.
Your migration budget covered the weekend. The real bill was hundreds of SQL queries, six DBAs, and six months.Share on X
The open checkbook
The problem was big enough, and the alternative, a rollback of the entire migration, expensive enough, that I effectively had an open checkbook.
I could spend whatever it took. What it took was a team of six database administrators, specialists in the new platform, working for roughly six months, and you have to understand that DBAs of that caliber aren’t cheap. Six of them, half a year, at emergency rates.
I hired a specialist firm instead of pushing the work onto internal staff, and the reasoning holds up decades later. Query optimization at that depth is a distinct craft: reading execution plans, restructuring joins, redesigning indexes for the new engine’s behavior. My people were experts in our systems; the firm’s people were experts in making this database fast. The combination worked. Query by query, hundreds of times over, the platform climbed out of barely tolerable and into healthy.
My take
Budget for the query layer before you sign off on the migration. Queries that test fine alone can collapse together under production load, and the bill shows up after go-live. Inventory the workload, benchmark the heaviest and most frequent queries against the new platform with production-scale data, and multiply across the inventory. That number is your real estimate. If the problem lands anyway, hire specialists in the new database. Ours took six DBAs about six months.
How should you budget for a database migration?
I tell executives planning a platform migration that their budget covers the visible work: conversion, cutover, infrastructure, testing.
The invisible line item is the accumulated tuning of every query the business runs, an asset that doesn’t transfer between platforms and has to be rebuilt on the new one. Price that rebuild before you commit, or price it afterward at emergency rates with the company standing on a struggling platform. We did it the second way, and I wouldn’t wish it on anyone.
The first way is available to anyone who asks the question in time, and the question is simple: what’s our plan and budget for re-optimizing the query layer, and who’s going to do it? A migration plan that skips that question has budgeted the weekend and left the rest of the migration to luck. I get angry every time I see one, because the people who pay for that gap are the employees stuck on a struggling platform for months while the fix gets bought at emergency rates. For more from this series, see The Digital Transformation Hub: real transformations, lived from the inside, decades before the term existed.
