A shipping platform must migrate years of order and tracking data to a new store. What strategy handles both the historical bulk and the ongoing changes?

The two-part strategy

A bulk backfill of history plus a change stream for ongoing writes, converging at a point where both are complete.

• Start the change stream first, capturing changes from a known position before the backfill begins. If the backfill runs first, changes during it are lost and the gap is silent.
• Backfill from a consistent snapshot, in batches, resumable and idempotent, with a bounded rate that does not degrade production.
• Apply the change stream from the snapshot's position, with idempotent handling since a change may be applied to a row the backfill has already written.
• Converge, when the stream lag reaches near zero and the backfill is complete.

The verification that is not optional

• Row counts and checksums by partition, which catches gross omissions.
• Field-level comparison on a sample, which catches transformation errors that counts cannot.
• Full reconciliation of anything financial, since a sample-level match implies an unknown discrepancy elsewhere and for money that is not acceptable.
• Comparison of derived values, not just stored ones — an aggregate that differs indicates a semantic difference that field comparison missed.

What causes migrations to fail

• Data that does not fit the new model. Years of a system's life include records created before a constraint existed, records fixed by hand, and encodings nobody documented. The transformation must have a quarantine path for records that cannot be converted, rather than failing the run or silently dropping them.
• Undocumented consumers discovered after cutover, reading the old store directly.
• An untested rollback. If verification fails after cutover, the ability to return to the old system must exist and must have been exercised.
• Underestimating the tail. The last 2% of records take most of the effort, because they are the anomalies.

The historical data question worth asking

Does all of it need to move? Migrating ten years of tracking data into an expensive new store when queries almost never reach beyond twelve months is a common and avoidable cost. Migrate the active window and archive the rest, with a documented path to retrieve it — which reduces the migration's size, duration and risk simultaneously.