intermediate 3 min answer

A CDC connector on a 4 TB Postgres table is restarted with a changed configuration during business hours, and the operator does not realise this triggers a new initial snapshot. What happens over the next three hours, upstream and downstream?

cdcsnapshotreplication slotbackpressurewal
Show the full answer Hide the answer

Second by second, what happens

The connector begins reading the whole table and republishing every row as a change event, while the live stream of new changes waits behind it in most connector designs. Three effects start at once.

  • On the source database, a long sequential read competes with the transactional workload for buffer cache and IO. The read itself is survivable; what usually is not is that the replication slot stops advancing while the snapshot runs, so the database retains every WAL segment produced since the snapshot began. Three hours of a busy write workload can be tens or hundreds of gigabytes of WAL that cannot be recycled, and the failure at the end of that road is a full disk on the primary.
  • On the topic, 4 TB of rows arrive as events at whatever rate the connector can manage, which is orders of magnitude above the normal change rate. Partitions grow, retention windows compress in wall-clock terms, and a topic sized for a day of changes now holds an hour.
  • On the consumers, lag climbs from seconds to hours. Every consumer sees each row as an update, so anything that acts on updates acts on the entire table: search reindexes everything, a notification service emails everyone, a downstream aggregation double-counts if it is not idempotent.

Where it amplifies

The most damaging amplification is at the edge of the system, not inside it. A consumer that treats the change stream as "something changed, tell the user" will send 4 TB worth of notifications. This is the failure that reaches customers, and it is entirely downstream of an operational action that looked like a config change.

What stops it

  • Snapshot modes, chosen deliberately. Most connectors offer a mode that skips the snapshot and resumes from the current log position, and incremental snapshot modes that interleave chunks of the table with live changes so the slot keeps advancing. Knowing which mode a restart uses is the operational fact that matters, and it is usually learned during the incident.
  • A monitor on replication slot lag in bytes, alerting well below the point where WAL retention threatens the primary's disk. This is the alert that would have caught it in minutes rather than hours.
  • Idempotent consumers and a snapshot marker on events, so consumers can distinguish "the row changed" from "the row was re-emitted by a snapshot" and suppress side effects.
  • Rate limiting the connector, accepting a longer snapshot in exchange for leaving the source and the topic usable.

What would have to be true for it to self-heal

Only that every consumer is idempotent, every side effect is suppressed on snapshot events, and the source has WAL headroom for the snapshot duration. Two of those three are design decisions made months earlier, which is why the answer to this question is mostly about what you built before the restart, not about what you do during it.

When this is not worth engineering around

On a table of a few gigabytes with no downstream side effects, a re-snapshot is a minute of extra load and nothing else. The protections above are justified by table size multiplied by what consumers do on an update, not by the presence of CDC. A reference table republished in full costs nothing; an orders table wired to a notification service costs a customer apology.