Change Data Capture Pipeline  ·  View 08 of 21  ·  Structure

Platform Components

Four planes in one region: capture, transport, projection, and a small exact control plane.

Editable source SVG draw.io All views
Google Cloud · europe-west1 Capture plane — reads the source, never writes it Log reader Datastream Snapshot chunker PK ranges Relation detector schema drift Column masker before the log Event publisher normalise + key Transport — the system of record for change Change log Pub/Sub · 7 days Dead-letter topic Changelog archive Cloud Storage Projection plane — one job per sink, one offset per sink Warehouse applier Dataflow Search indexer Dataflow Fan-out worker Cloud Run Rebuild runner shadow then swap Control plane — small, exact, authoritative Stream and sink registry Spanner Schema registry versioned Operator API Cloud Run Reconciliation runner daily Operational PostgreSQL 12 databases BigQuery Search index WAL publish by offset MERGE schema version Platform Components — Container View Interface / broker Application we own Security / platform Queue / topic Data store External / third party synchronous event / async batch Omitted for clarity: the search and fan-out sink edges, Secret Manager and IAM into capture, and every metrics edge. v 1.0 · owner Data Platform Architecture · date 2026-10

Decisions

  • Control state lives in Spanner, apart from every projection: a few million exact rows and billions of rebuildable ones have different recovery stories (ADR-14).
  • One Dataflow job per sink, each with its own offset, so a sink's failure is a sink's failure.
  • The rebuild runner is a first-class component, not a script: rebuild is the primary recovery path for every sink (ADR-11).

Stack

  • Datastream for log-based capture; Pub/Sub for the change log; Dataflow for transform and apply; Cloud Run for the operator API; Spanner for control state.
  • The requirement stays vendor-neutral: every one of these has an equivalent on two other clouds and in open source (see the stack table).

Omitted

  • Search and fan-out sink edges, Secret Manager and IAM into capture, and all metrics edges.
◀  Layered Architecture All views Integration Surface  ▶
Solution Architecture v1.0 · Google Cloud · Data Platform Architecture · 2026-10 · 21 views · 15 architecture decision records08-platform-components