Change Data Capture Pipeline  ·  View 12 of 21  ·  Data

Control-Plane Data Model

Ten entities, and the composite key that makes an apply idempotent.

Editable source SVG draw.io All views
source_database source_id PK engine replication_slot log_retention_hours source_budget_pct replicated_table table_id PK source_id FK -> source_database tier (1 | 2) pre_image_available state (live | paused | snapshotting) schema_version schema_version_id PK table_id FK -> replicated_table effective_lsn columns JSON compatibility (compatible | breaking) sink sink_id PK shape (mirror | changelog) freshness_target_s delete_mode (soft | hard) sink_subscription subscription_id PK sink_id FK -> sink table_id FK -> replicated_table offset_lsn lag_seconds snapshot_chunk chunk_id PK table_id FK -> replicated_table pk_range_low pk_range_high state (pending | done) change_event table_id + pk + lsn PK op (c | u | d | r) schema_version_id FK commit_ts post_image JSON dead_letter_event dlq_id PK table_id FK -> replicated_table lsn error_class payload raw reconciliation_run run_id PK subscription_id FK -> sink_subscription verdict (match | diverged) rows_source / rows_sink sampled_key_ranges operator_action action_id PK actor action (pause | resume | resnapshot | reset) target_sink_id FK -> sink target_table_id FK -> replicated_table at immutable 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N Control-Plane Data Model change_event is the log's shape, not a table: it is keyed by (table, primary key, log position) so an apply is idempotent. v 1.0 · owner Data Platform Architecture · date 2026-10

Decisions

  • change_event is the log's shape rather than a table, keyed by (table, primary key, log position) — the key that makes a replay a no-op (ADR-04).
  • schema_version carries the log position at which it became effective, so an event and its schema are matched by position, not by time.
  • snapshot_chunk exists so a snapshot is resumable; without it a failure at 94% is a restart (ADR-05).

Assumptions

  • Tier is 1 or 2 and decides reconciliation frequency and alarm severity; tier-1 is assumed to be 40 tables.
  • delete_mode is per sink and per table, defaulting to soft (ADR-10).
  • pre_image_available is a recorded property of the table, not an assumption of the pipeline.

Omitted

  • Sink-side schemas, which are generated from schema_version rather than modelled here.
  • operator_action's table target is drawn as optional; the model carries both foreign keys.
◀  Storage Zones All views Critical Flow — Commit to Sink  ▶
Solution Architecture v1.0 · Google Cloud · Data Platform Architecture · 2026-10 · 21 views · 15 architecture decision records12-data-model