Change Data Capture Pipeline  ·  View 04 of 21  ·  People and journeys

Journey — Wire Up a New Sink

The journey the platform exists to make cheap, and the one phase where it is still feared.

Editable source SVG draw.io All views
Data engineer 6 on the team Goal — Give the new fraud service a current-state mirror of 9 tables Trigger — A product team asks for data that already exists somewhere Done when — Sink registered, backfilled and inside its freshness target, with no source ticket 1 · Ask before any code 2 · Register 3 · Backfill ◆ moment of truth 4 · Catch up 5 · Operate What they do Names the tables Declares freshness Registers the sink Picks mirror or log Starts the snapshot Watches lag fall Subscribes to alarms What the platform does Shows cost per table Allocates an offset Creates sink schema Chunks by PK range Throttles to source budget Replays log over snapshot Publishes freshness How it feels Confident Fine Anxious Where it hurts Backfill ETA unknown Fear of hurting the source What answers it Per-table cost report Self-service registry Chunk progress and ETA Declared source budget Lag graph per sink Freshness as a contract Journey — Data Engineer Wires Up a New Sink The trough is the backfill: an unbounded snapshot against a live primary is the moment this job is feared. v 1.0 · owner Data Platform Architecture · date 2026-10

The trough is the backfill

  • The low point is not registration; it is pressing go on a snapshot of a 400 GB table against a live primary with no ETA.
  • The architecture's answer is structural: chunked snapshot by primary-key range, a declared source budget, and chunk-level progress — so the operation is interruptible, resumable and bounded in its impact (ADR-05, ADR-06).

What makes the rest easy

  • Self-service registration against the log: no capture change, no source ticket.
  • Per-table cost shown before the sink is registered, so an unused table is a decision rather than a surprise.
  • Freshness published per sink, so the sink owner reads a contract instead of asking.

Assumptions

  • Largest single table 400 GB / 1.2 billion rows; snapshot ≥ 150 GB/hour at the declared budget.
  • A nine-table sink is onboarded in a day of engineer time, excluding backfill wall-clock.
◀  Actors and Their Journeys All views Journey — Ship a Schema Migration  ▶
Solution Architecture v1.0 · Google Cloud · Data Platform Architecture · 2026-10 · 21 views · 15 architecture decision records04-journey-wire-up-a-new-sink