A platform builds change data capture from its operational databases. What is the failure that turns an analytics problem into a production incident?

The failure

Replication slot backpressure onto the source database.

Some databases retain replication log segments while a consuming slot is behind. A stalled CDC connector therefore causes the primary's disk to fill — and the analytics pipeline takes down the transactional system.

This is the one failure mode that must be designed for explicitly, with a disconnect policy: if the connector falls behind by more than a defined threshold, it is disconnected and a re-snapshot is scheduled, rather than allowing unbounded retention.

The other principal failure modes

1. Schema evolution. A column dropped or retyped upstream breaks consumers. The database schema has become a public interface whether or not the owning team agreed — so migrations must be reviewed as contract changes, backed by a registry with compatibility rules.

2. Log retention shorter than the worst realistic outage. If the connector cannot resume, a full re-snapshot is required — expensive and disruptive. Retention must exceed the worst outage, and lag must alert well before that horizon.

3. Duplicate delivery, since at-least-once means consumers must upsert by primary key rather than insert.

4. Transaction boundaries lost. Changes to several tables in one transaction arrive as separate events, possibly in different partitions, so a consumer can observe a state that never existed atomically. If cross-table consistency matters, the consumer must group by transaction identifier and apply atomically — and many implementations quietly do not.

5. The snapshot-to-stream handover, which is the fiddliest part of any implementation: a new consumer needs a consistent initial snapshot and then the stream from exactly that log position.

Why log-based beats the alternatives

Query-based polling adds load to the primary, misses deletes entirely, misses intermediate states between polls, depends on a correctly-maintained timestamp column, and races at the cursor boundary.

Application dual-write has no atomicity — a crash between the two writes diverges the systems permanently with no mechanism to detect it.

The coupling to manage

Downstream systems now depend on an operational database's internal schema. Publish a contract — a view or a projection — as the interface, so the owning team can refactor underneath without breaking consumers they have never met.