Change Data Capture Pipeline · View 10 of 21 · Data
The three fields that matter
- Post-image, primary key and log position: idempotence, ordering and staleness detection all derive from these.
- The schema version is stamped at capture so a replayed event is interpretable years later (ADR-08).
- Tenant id is derived at capture and never inferred at the sink, because an inferred tenant is a cross-tenant leak waiting for a refactor.
Assumptions
- Micro-batch flushes at 5 seconds or 10,000 rows, whichever comes first.
- p99 event ≤ 64 KB; events over 1 MB are dead-lettered with a size error.
- 780 million change events a day, about 1.1 TB uncompressed.
Risks
- Masking at capture is irreversible by design: a column excluded here cannot be backfilled without a re-snapshot.
- Discarding a stale event depends on a monotonic sequence per key; a sink that ignores it will overwrite new values with old ones.
