Change Data Capture Pipeline  ·  View 14 of 21  ·  Runtime

Snapshot to Stream Handoff

The initial-snapshot problem, solved with no lock and no gap.

Editable source SVG draw.io All views
Register Table registered allow-list applied Slot position recorded LSN₀ Stream first Stream from LSN₀ into the log Sink stays empty offset parked Chunked snapshot Next PK range resumable Read from replica op = READ Yield to live traffic ≤ 5% source CPU Overlap resolved Replay log over snapshot from LSN₀ Keep the higher LSN per primary key Steady state Table marked live one apply path Lag inside target p95 ≤ 5 s If it breaks Resume at last chunk never restart Log truncated: re-snapshot fail loudly LSN₀ chunk state Snapshot to Stream — The Handoff With No Gap Application we own Decision point Queue / topic Interface / broker Security / platform Opportunity Risk / gap synchronous Snapshot rows and stream rows are the same event type with a different op, so one apply path serves both. v 1.0 · owner Data Platform Architecture · date 2026-10

The mechanism

  • Record the log position first, stream into the log from that position, then snapshot in primary-key chunks and replay the overlap on top (ADR-05).
  • Correctness rests on one property: the apply path keeps the higher log position per key. Snapshot rows carry op = READ and the lowest precedence.
  • Snapshot and stream are the same event type on the same apply path, so backfill cannot diverge from live capture.

Alternatives rejected

  • Lock the table and take one consistent position — correct, and unacceptable on a live primary.
  • Snapshot a point-in-time clone — clean seam, but it ties the design to the engine's cloning and pays for a clone per table (see ADR-05 options).

Failure handling

  • An interrupted snapshot resumes at the last completed chunk; it never restarts.
  • If the source log is truncated while a snapshot runs, the table is marked for re-snapshot and fails loudly — never resumed with a gap (ADR-03).
◀  Critical Flow — Commit to Sink All views Schema Drift — Four Paths  ▶
Solution Architecture v1.0 · Google Cloud · Data Platform Architecture · 2026-10 · 21 views · 15 architecture decision records14-snapshot-to-stream-handoff