Change Data Capture Pipeline  ·  View 01 of 21  ·  Context and scope

System Context

One pipeline between one writer and many readers — and the three dependencies that can stop it.

Editable source SVG draw.io All views
Systems of record (out of scope) Operational PostgreSQL 12 DBs · 180 tables Read replica snapshot source People Data engineer Product engineer Platform SRE Analyst Sinks the pipeline writes BigQuery mirror + changelog Search index product search Cache and webhooks fan-out CDC Pipeline commit to sink ≤ 5 s p95 Platform dependencies Cloud IAM workload identity Secret Manager 90-day rotation Cloud Storage 13-month archive Cloud Monitoring lag and slot SLOs WAL decode snapshot read registers sinks ships migrations pauses, replays queries upsert index upsert invalidations authorises credentials archive lag, alarms Change Data Capture Pipeline — System Context External / third party Person or role Data store Security / platform synchronous event / async batch The pipeline owns no business logic: it owns the seam between one writer and many readers. v 1.0 · owner Data Platform Architecture · date 2026-10

Decisions

  • The source databases are out of scope and read-only to this platform: capture holds a replication identity with no write privilege.
  • Sinks are inside the boundary as projections, but their consumers are not: the pipeline owes them freshness, not query semantics.
  • The heartbeat row written into every source database is part of the pipeline, not of the application — it is how lag is measured on an idle table.

Assumptions

  • 4,000 tenants across 12 operational PostgreSQL databases and 180 replicated tables.
  • 9,000 row changes a second at steady state, 36,000 for 30 minutes during a bulk tenant import.
  • Three sink families today — warehouse, search, cache and webhook fan-out; a fourth is expected within a year.

Risks

  • The replication slot is a shared fate with the source: an abandoned slot fills the source's disk, which is an application outage caused by this platform.
  • Consumers reading the mirror without the as-of field will report staleness as wrongness.
All views High-Level Architecture  ▶
Solution Architecture v1.0 · Google Cloud · Data Platform Architecture · 2026-10 · 21 views · 15 architecture decision records01-system-context