Snapshot-Stream Convergence

Populating a destination from a source requires both the existing data and the ongoing changes. The order in which they are started determines whether there is a gap.

Start the change stream first, from a known position. Then snapshot. Then apply the stream from that position, idempotently.

If the snapshot runs first and the stream starts afterwards, every change that occurred during the snapshot is lost — silently, with no error, and the destination is permanently wrong for the affected rows.

Why it matters

The gap is invisible. Row counts match, the pipeline reports success, and a subset of records reflects a state from partway through the backfill. It is discovered — if at all — by a reconciliation weeks later, and it is very difficult to attribute after the fact.

Implementation patterns

• Record the log position before starting the snapshot, and apply from exactly that position afterwards.
• Idempotent application, since the stream will replay changes to rows the snapshot has already written. An upsert keyed on the primary key handles it; an insert does not.
• Batched, resumable, rate-limited snapshot, since it will be interrupted and it must not saturate the source.
• Handle deletes explicitly. A hard delete in the source must be represented, and a destination that ignores deletes silently accumulates records that no longer exist — producing counts that drift upward and reports that are wrong in a direction nobody checks.
• Preserve per-key ordering through the consumer, which is free from a partitioned log keyed on the primary key and is broken if the consumer processes a partition concurrently.
• Verify by reconciliation, not by row count alone: counts and checksums by partition catch gross omissions, and field-level comparison on a sample catches transformation errors that counts cannot.

Industry example

Logistics platforms such as Delhivery feed warehouses from operational databases whose rows change constantly during a multi-hour backfill. The ordering discipline is what makes the initial load correct, and it is the kind of detail that is obvious in retrospect and routinely got wrong the first time.

The same pattern applies to any migration with a bulk phase and an ongoing phase — a warehouse migration, a service extraction, a search index rebuild.

Failure scenarios

• Snapshot before stream, losing every change made during the backfill.
• Non-idempotent application, producing duplicates where the snapshot and stream overlap.
• Deletes ignored, accumulating phantom records.
• Concurrent consumer processing, discarding the ordering the partitioning provided.
• Verification by row count alone, which passes while values are wrong.

Trade-offs

Starting the stream first means buffering changes for the snapshot's duration, which for a long backfill is substantial storage and requires the log's retention to exceed the snapshot's duration — a constraint that is easy to violate on a large table and produces a failure at the worst moment.

The alternative is a source-side lock or a consistent snapshot mechanism, which is cleaner where the source supports it and unavailable on many systems — which is why the ordering discipline plus adequate log retention is the general answer.

Interview question

"You are backfilling a hundred-million-row table into a new store while it is being written to. Tell me the exact order of operations, and what breaks if you get it wrong."