A product must move its primary data store while remaining live, with no acceptable data loss. What is the migration sequence, and where do these go wrong?
Show the full answer Hide the answer
The sequence
1. Provision the target and replicate historical data, without any live traffic. Establishes the target's operational maturity — monitoring, capacity, failover — before anything depends on it.
2. Dual-write. The application writes to both stores, with the old one authoritative. The new store now receives current changes while the backfill catches up on history.
3. Backfill history in batches, rate limited, so the migration does not become a load incident on the source.
4. Verify. Compare row counts, checksums and sampled records between the stores, continuously. This is the step most often shortened and the one that prevents the worst outcome — most migration disasters are discovered after the old data is gone.
5. Dual-read with comparison. Read from both, serve the old, log discrepancies. This turns correctness into a measurement rather than a hope, and it catches the differences that count-and-checksum comparison misses — ordering, encoding, precision, null handling.
6. Cut reads over incrementally, a percentage at a time, with instant rollback as a configuration change.
7. Make the new store authoritative, then stop writing to the old.
8. Decommission, deliberately, after a retention period during which the old store remains restorable.
Where these go wrong
Cutover granularity. Cutting over globally means the blast radius of a mistake is every customer. Cutting over per tenant, per workspace or per shard means a mistake affects one customer and can be reverted — and this single choice is the largest risk reduction available.
Dual-write consistency. Writing to two stores is not atomic. A failure between them diverges the data, and without reconciliation the divergence is silent. Either write to the source of truth and derive the other asynchronously from its change log, or accept dual-write with continuous reconciliation and alerting.
Verification that only counts rows. Equal counts with different contents is a common and quiet failure.
No rollback after authority moves. Once the new store is authoritative and has taken writes the old one has not, rollback means losing those writes. The plan must state where the point of no return is, and what the recovery is beyond it.
Backfill starving live traffic, which turns a background migration into a production incident.