By Prajakta Shende — Database Customer Engineer at Google Cloud, with close to a decade at AWS in the data space before that.
Every migration plan works on the whiteboard. I've spent a decade moving production databases — petabytes of them, across engines, clouds, and more cutover nights than I care to count — and I can tell you exactly where the plan meets reality. Petabytes don't break migrations in random places. They break them in five predictable ones, in roughly this order.
1. The throughput math nobody does
Before any dates get committed, three numbers decide your migration: how fast the source can be read, how fast the network can move data, and how fast the target can write. Your actual migration speed is the slowest of the three — and it's almost never the one the plan assumed.
The second mistake is treating the bulk copy as the schedule. It isn't. The cutover window is set by change-data-capture lag: how long it takes the target to replay every change that landed during the copy. On a quiet system that's minutes. On a system writing thousands of transactions a second, an untested CDC pipeline can fall hours behind — and you find out during the cutover window you promised the business.
Do the math before you promise the date. Model the backfill, measure the change rate at peak (not average), and rehearse the catch-up. If you can't state your expected CDC lag at cutover within an order of magnitude, you don't have a plan yet. You have a hope.
2. Schema drift and the long tail of types
Automated schema converters are wonderful liars. They'll convert 95% of your tables cleanly and report success — and the remaining 5% will eat your entire schedule.
The failures are rarely dramatic. They're semantic: a timestamp type that silently drops timezone behavior, a collation difference that changes sort order and breaks a dedup query, numeric precision that rounds differently in the new engine. Nothing crashes. The data is just wrong, quietly, in ways that surface weeks later in a finance report.
At petabyte scale you cannot eyeball this. You need row counts, checksums, and — hardest of all — a deliberate inventory of every engine-specific type and function in the source, reviewed by a human who knows both engines, before conversion starts. The converter is a tool. The inventory is the work.
3. The dual-write trap at cutover
Dual-write feels safe: write to both systems, compare, switch over when they match. In practice, dual-write is where consistency goes to hide. Writes fail asymmetrically. Retries duplicate. One system applies an update the other rejected, and the divergence is invisible until reconciliation — if reconciliation is even looking at the right tables.
And reconciliation at petabyte scale has its own trap: full-table validation costs real time and money, so teams sample. Sampling finds the errors you expected. It misses the one corrupted partition you didn't.
The middle path that actually works: checksum validation on the tables that matter most (money, identity, state), row-count and aggregate validation everywhere else, and — this is the part teams skip — rollback criteria written down before cutover night. At 3 a.m., nobody makes good decisions about whether to push forward or roll back. Decide at 3 p.m. the week before, in writing, with numbers.
4. Post-migration performance regressions
The migration "succeeds." The data is there. And then the p99 latency graph starts climbing.
Every long-lived database carries years of tuning debt in the best sense: query plans the old optimizer had settled into, statistics gathered over years, access patterns the application grew around. The new engine starts from zero. Its optimizer makes different choices — sometimes better, often just different — and different loses to familiar when a checkout flow is timing out.
The fix is unglamorous and systematic: capture the top queries by business impact (not by count), compare plans old versus new, rebuild statistics deliberately instead of waiting for auto-analyze, and work the list in impact order. I've seen this work recover 35% performance improvements over the pre-migration baseline once the tuning debt got repaid in the new engine's currency — but only for teams that budgeted the weeks to do it. If your migration plan ends at cutover, your plan ends too early.
5. The cost surprises
Finally, the bill. Migrations carry tail costs nobody modeled: replication traffic, cross-region transfer during months of parallel running, the old engine's license tail, and target storage replicated three ways because nobody revisited the defaults.
The real savings come later, and they're structural. In my own cost-optimization work, the 40% reduction in managed database spend didn't come from the migration itself — it came from the rightsizing pass after the dust settled: instances sized for the migration's peak rather than production's steady state, storage tiers revisited, reserved capacity bought once the new shape of the workload was known. Budget that pass into the plan, or the "savings" slide in the kickoff deck stays fiction.
The pre-flight checklist
Before any petabyte-scale migration, I want written answers to five questions:
- What is our measured bottleneck — source read, network, or target write — and what's the number?
- What is our expected CDC lag at cutover, at peak change rate, and how did we rehearse it?
- Where is the human-reviewed inventory of engine-specific types and functions?
- What are the rollback criteria, in numbers, agreed before cutover night?
- What does the post-migration tuning and rightsizing pass look like, and who owns it?
Migrations don't fail because teams lack tools. They fail because scale turns small unknowns into schedule-ending surprises. The five places above are where those surprises live. Go look there first.
