Idempotent Replay

A migrator copying trillions of rows will crash. The disk will fill, a dependency will time out, a deployment will restart the pod, the cluster will be rebalanced. The question that decides whether that is a five-minute event or a three-week investigation is whether a unit of work can simply be run again.

Idempotent replay is the property that makes it the five-minute event: each unit writes with the source's own keys and timestamps, has no side effects, and overwrites rather than appends, so re-running it converges on the same state.

Why it matters

Without it, every failure produces a region of the dataset in an unknown condition, and the only honest recovery is to compare the two systems over that region - which is a migration-sized task performed under pressure. The cost of non-idempotency is not the crash; it is the loss of the ability to trust the data afterwards.

It also changes the migration's operating model. Idempotent units can be run in parallel without coordination, re-run selectively after a bug fix in the transform, and re-run entirely as a final verification pass. A migrator you can safely run twice is one you can safely run at all.

Implementation patterns

• Carry the source identity. Use the source's primary key as the target's, or a deterministic function of it. Generating new identifiers is the most common way a backfill becomes non-replayable.
• Carry the source timestamp. In a last-write-wins store, writing with the migrator's clock lets stale values win a later comparison and resurrects superseded data.
• Write, do not increment. Any counter, append or event emission inside the unit breaks replay. If downstream notification is needed, derive it from the target store afterwards rather than from the act of migrating.
• Choose a unit that maps to a natural partition - one tenant, one shard, one hash range, one day - and record a watermark per unit, so progress is restartable at that granularity.
• Make deletes explicit. Tombstones and soft-deleted rows must be part of the unit's definition, or replay re-animates deleted data.
• Guard with a conditional write where the store supports it (write only if the target version is older), which makes replay safe even when the source has moved on.
• Run the whole thing twice deliberately before cutover. A clean second pass with a zero diff is the cheapest proof of correctness available.

Industry example

Discord's 2023 account of moving its message store describes a purpose-built migrator in Rust, reported as moving the bulk of trillions of rows in around nine days, with per-partition progress and the ability to re-drive partitions - the shape every large migration converges on. The same discipline is standard in change-data-capture pipelines, where at-least-once delivery is guaranteed and exactly-once is not, so every consumer must be able to apply the same change twice. Apache Hudi and similar table formats expose upserts for precisely this reason: a change-data-capture pipeline that re-delivers yesterday's batch must converge rather than duplicate.

Failure scenarios

• New ids generated per run, so a re-run duplicates every row it touches.
• Migration-clock timestamps, after which a legitimate older write from the source appears newer and wins.
• Events emitted per migrated row, so a re-run sends a million duplicate notifications to customers.
• A global watermark instead of per-partition, so one failure restarts the entire dataset.
• Deletes handled outside the unit, leaving resurrected records that reconciliation finds months later.
• Non-deterministic transforms - a lookup against a service whose answer changes, or a now() in the mapping - which make two runs legitimately disagree.

Trade-offs

Idempotency constrains the design: you give up generated identifiers, in-band event emission and append-style writes, and you accept the overhead of conditional writes or version checks. For a small one-off copy that is real work for no benefit. What it buys is operational freedom - parallelism without coordination, selective re-runs after a transform bug, and a final verification pass - which on any dataset large enough to take days is worth more than the throughput it costs.

When not to use it

For a dataset that copies inside a maintenance window, a stop-the-world copy needs none of this: nothing is concurrent, nothing crashes for long, and a failed attempt is simply repeated from the start. It is also the wrong frame when the migration is inherently a transformation with side effects by design - for example a one-time backfill that must emit a welcome notification per migrated account - in which case the side effect should be moved out of the unit and driven separately from the target state, rather than making the unit non-replayable.

Interview question

Q: Your backfill crashed 60% of the way through a 4 TB table. The team wants to know whether to resume or start again. What determines the answer, and what would you have built differently?

What a strong answer covers: that resumption is safe only if units were idempotent and watermarks per-partition; checking for generated ids, migrator-clock timestamps, emitted events and non-deterministic transforms, since any of those makes the already-copied region untrustworthy; proposing a verification pass over the region rather than a guess; and the redesign - source keys and timestamps, conditional writes, per-partition watermarks, deletes inside the unit, and a deliberate second full pass with a zero-diff target before cutover.

Quick check

Quiz: Why must a backfill write the source's original timestamps? In a last-write-wins store a migrator-clock timestamp makes copied data appear newer, so later legitimate updates and deletes can be lost or reversed.

Flashcard: Three things that make a backfill unit non-replayable? Generated identifiers, writes stamped with the migrator's clock, and side effects such as emitting an event or incrementing a counter per row.