CDC Pipeline Design
The concerns that turn log-based change capture from a demo into something that can be relied on — snapshots, ordering, schema change and replay.
CDC is straightforward until it meets reality, and the difficulties are consistent across tools.
Initial snapshot and cutover. The stream starts from now; the table already has history. Taking a consistent snapshot while capturing ongoing changes, then stitching the two without gaps or duplicates, is the part that goes wrong and the part vendors gloss over.
Ordering. Correctness depends on applying changes in order per key. Partitioning the stream by primary key preserves that; partitioning any other way produces an update applied before the insert it depends on, sporadically, under load.
Schema evolution. A column added upstream on a Tuesday afternoon must not stop the pipeline. This requires a schema registry with a compatibility policy and a defined behaviour for incompatible changes, agreed with the source team in advance rather than discovered during an incident.
Replication slot management. A stalled consumer causes the source database to retain write-ahead log segments indefinitely, and the failure mode is the source system's disk filling — a data platform outage that becomes a production database outage. This needs its own alert, and it is routinely missing.
Replay. Being able to reprocess from a retained log is what makes downstream mistakes recoverable, and retention must be sized for that deliberately.