Change Data Capture Pipeline  ·  View 21 of 21  ·  Assurance

Failure Classes

Nine classes, their detection, their blast radius — and no recovery that is hand-written SQL.

Editable source SVG draw.io All views
Detected by Contained to Recovery Correctness risk Transient (blip, throttle) Retry counters One batch Backoff with jitter None — lag absorbs it Source unavailable Connection loss One source Resume at last LSN None Source log truncated Position gap slot headroom alarm One table Re-snapshot fail loudly first Silent gap if unchecked Source failover / new timeline LSN discontinuity One source Reconcile or re-snapshot Bounded duplicate or gap Breaking schema change DDL event One table slot keeps running Operator decision None if paused Poison event Retry budget spent One event Dead-letter with context One row stale Sink unavailable or slow Apply lag per sink That sink only Buffer in the log to 7 days Past retention: re-snapshot Bad transform deployed Reconciliation verdict That sink only Replay to shadow, swap Wrong values until found Regional outage Health checks Whole pipeline Standby from Spanner LSN RTO 30 min None — RPO 0 vs source Assurance — Failure Classes and Their Handling Only one row has a whole-pipeline blast radius, and no row has a recovery that is hand-written SQL. v 1.0 · owner Data Platform Architecture · date 2026-10

The pattern

  • Eight of nine classes are contained to one event, one table or one sink. Only a regional outage has a whole-pipeline radius (ADR-15).
  • Every recovery is resume, replay, rebuild or re-snapshot. A class whose recovery is a human writing an UPDATE is a design defect, not an incident.
  • Two classes carry a correctness risk that detection alone does not remove: a truncated log and a source failover timeline.

Assumptions

  • Zero acknowledged-change loss; duplicate rate ≤ 0.1% of delivered events.
  • Per-row commit ordering preserved 100%; no ordering guarantee is offered across rows or tables.

The honest gap

  • A bad transform is detected by reconciliation, which means it is detected in hours rather than seconds — the cheapest improvement available to this design is reconciliation coverage, not faster capture.
◀  Identity and Access Flow All views
Solution Architecture v1.0 · Google Cloud · Data Platform Architecture · 2026-10 · 21 views · 15 architecture decision records21-failure-classes