Replication Slot Retention

A change-data-capture pipeline reading from a database's write-ahead log is not a passive observer. The slot it reads from is a promise by the database to keep every log segment until the consumer confirms it has processed them. That promise is what makes CDC reliable: a connector can be down for an hour and resume without losing changes.

It is also a coupling that points the wrong way. A consumer problem becomes a database problem, because the log the consumer has not read cannot be recycled, and the disk it sits on belongs to the primary that is serving production traffic.

Why it matters

Most teams model the CDC pipeline as downstream and therefore harmless: if it breaks, data arrives late. The retention mechanism inverts that. A connector that is stopped for a weekend, stuck retrying a poison record, or running a three-hour snapshot during which the slot does not advance, accumulates WAL at the rate the database produces it. A busy transactional database producing 50 GB of WAL a day fills 150 GB of headroom in three days of a paused connector.

The failure at the end is not degraded replication. It is a full disk on the primary, which stops writes entirely. The blast radius of a forgotten connector is the whole application.

Implementation patterns

• Alert on slot lag in bytes, not on connector health. The connector can be running and the slot still not advancing, during an initial snapshot or when a consumer's commits are failing. The byte figure is the only signal that tracks the actual risk.
• Set the threshold from disk headroom, not from a round number. The useful alert is "slot lag exceeds 20% of free space on the WAL volume", which scales with the instance.
• Use a bounded maximum where the engine supports it (Postgres offers a cap on slot-retained WAL), accepting that exceeding it invalidates the slot and forces a re-snapshot. That is a deliberate choice of which failure you prefer: a pipeline that must rebuild, or a primary that stops accepting writes.
• Prefer incremental snapshot modes, which interleave table chunks with live change streaming so the slot keeps advancing during a snapshot rather than freezing for its duration.
• Delete slots as part of decommissioning. An orphaned slot from a retired pipeline is the classic cause: nothing is consuming it, nothing alerts on it, and it retains WAL forever.

Industry example

Logical replication slots have behaved this way in PostgreSQL since they were introduced in 9.4 (2014), and the equivalent mechanisms exist in MySQL binlog retention and SQL Server CDC capture jobs; each engine's operational guidance documents it. Connector projects in the Debezium family warn about it directly, because the most common production incident reported against log-based CDC is not lost data, it is WAL growth on the source while a connector is stopped or snapshotting.

Failure scenarios

• The three-hour snapshot. A connector restarted with changed configuration re-snapshots a 4 TB table; the slot freezes for the duration and tens to hundreds of gigabytes of WAL accumulate.
• The forgotten slot. A pipeline is decommissioned, the slot is not dropped, and the disk fills weeks later with no active connector to blame.
• The poison record. A connector crashes repeatedly on one message, restarts, and never commits an offset, so the slot advances not at all while the process looks alive.
• The failover surprise. After a failover, slots may not exist on the new primary depending on version and configuration, so the pipeline silently needs a full re-snapshot at the worst possible time.

Trade-offs

Retention is the price of reliable change capture. Removing it means accepting gaps: a connector that resumes at the current log position loses everything produced while it was down, which for most CDC use cases is unacceptable because downstream state becomes permanently wrong rather than late. The bounded-retention setting is the middle position, and it converts a database outage into a re-snapshot, which is itself expensive in load on the source and amplification downstream. There is no configuration in which a stopped consumer costs nothing.

When not to use it

When the source database has no headroom to spare and the data can tolerate gaps, log-based CDC is the wrong mechanism. A timestamp-based incremental query, an outbox table written by the application, or event publication from the service itself all avoid coupling the database's disk to a consumer's health. The outbox pattern in particular trades a small write amplification on every transaction for the removal of this entire failure class, and for many teams that is the better bargain. Choose log-based CDC when completeness matters more than isolation, and an outbox when isolation matters more than capturing changes the application did not intend to publish.

Interview question

Q: Your Postgres primary's disk is filling at 8 GB an hour and application writes are fine. Replication to the read replica is healthy. Where do you look, what do you do in the next ten minutes, and what do you change afterwards?

What a strong answer covers: checking slot lag per slot rather than replica lag, since the two are different mechanisms · identifying the slot that is not advancing and whether its consumer exists at all · the immediate choice between fixing the consumer and dropping the slot, with the consequence of each (a re-snapshot versus a stopped primary) · afterwards: byte-based alerting tied to headroom, bounded retention, incremental snapshots, and slot cleanup in the decommissioning checklist.

Quick check

Quiz: Why can a CDC connector be running and the slot still not advance? Because advancing requires confirmed processing; during an initial snapshot, or while a consumer fails to commit offsets, the connector is alive and confirming nothing.

Flashcard: What is the end state of an orphaned replication slot? — WAL retained indefinitely until the primary's disk fills and writes stop.