Change Data Capture and the Outbox Pattern

An order service commits a row to Postgres, then publishes OrderPlaced to Kafka. Between the two calls the pod is killed by a deploy. The order exists; the warehouse, the fraud model and the feature store never hear about it. Swap the order of the calls and the failure flips: an event announces an order the database rolled back. Suppose the gap between the two writes is hit once per 100,000 requests (an illustrative rate, from crashes, timeouts and broker blips). At 50 million orders a day that is 500 silent inconsistencies daily, and nothing in either system reports them.

The dual-write problem

Two independent systems offer no shared transaction. Distributed commit protocols exist, but many brokers and managed databases do not participate in them, and the ones that do pay in latency and availability. Kleppmann, Beresford and Svingen describe the practical failure of distributed transactions across heterogeneous storage, and argue for a design in which one append-only log is the single source that every other system derives from (Kleppmann et al., 2019, Online Event Processing, CACM 62(5)).

Atomicity is only half of it. With two concurrent writers, the database can commit \(A\) then \(B\) while the broker receives \(B\) then \(A\). Every consumer that applies the stream in order now ends in a different final state from the database, permanently, and no retry fixes an ordering that was never shared. The only robust answer is to write once, to one system, and derive the rest from its log.

Log-based capture

A relational database already keeps an ordered record of every committed change: the write-ahead log in Postgres, the binlog in MySQL. Log-based CDC reads it as a replication client. For Postgres that means a logical replication slot, which streams decoded row changes in commit order and remembers the reader's position. Debezium, a widely used open-source implementation, emits each change as an event with before and after images plus source position, and on restart resumes from the last recorded position (Debezium PostgreSQL connector documentation).

Compare the alternative, polling WHERE updated_at > :last_seen every \(\Delta\) seconds. A row updated \(n\) times within one interval surfaces once, so \(n-1\) intermediate states vanish. Hard deletes never surface at all. Clock skew between writers and the poller skips rows whose timestamp lands just behind the high-water mark. Log-based capture sees every committed transaction, deletes included, and adds no query load.

Existing rows need a snapshot before streaming begins, and a long locking snapshot of a large table is unacceptable on a live primary. Netflix's DBLog interleaves chunked SELECTs with the log stream, using watermark rows written to the log to decide which selected rows are superseded by concurrent changes, so full-state capture runs without locks and can pause and resume (Andreakis & Papapanagiotou, 2020, DBLog: A Watermark Based Change-Data-Capture Framework, arXiv:2010.12597). Debezium's incremental snapshots use the same low and high watermark technique.

The outbox: capture a contract, not a table

Raw CDC publishes the service's internal tables. Every column rename becomes a breaking change for consumers the service owner has never met, which is exactly the obligation data-contracts-as-producer-obligations says a producer must own deliberately.

The transactional outbox fixes this. In the same local transaction as the business write, the service inserts a row into an outbox table holding an event id, an aggregate type, an aggregate id and a payload shaped for consumers. Because it is one transaction, the event exists if and only if the business change committed. CDC then reads only the outbox. Debezium's outbox event router routes each row to a topic named from aggregatetype and keys the message by aggregateid, so all events for one order land in one partition and keep their order (Debezium Outbox Event Router).

Practitioners genuinely disagree on the default. Raw CDC needs no application change and captures everything, including writes from scripts and migrations that bypass the service. The outbox gives a stable, intentional contract but captures only what the code remembers to write. Many platforms end up running both: raw CDC into the lake for analytics, outbox events for service-to-service integration.

When it breaks

Delivery is at least once. After a connector crash, events since the last committed offset are re-emitted. Consumers must deduplicate on the event id or apply changes idempotently; the precise guarantees are the subject of exactly-once-semantics-precisely.

A stalled connector can fill the primary's disk. A replication slot prevents Postgres from discarding WAL the slot has not consumed. A connector down over a weekend on a busy database holds every WAL segment since Friday. PostgreSQL 13 added max_slot_wal_keep_size to cap this, trading a full disk for an invalidated slot and a forced re-snapshot.

Before images may be incomplete. With Postgres's default replica identity, update and delete events carry only primary-key columns in the before image. Consumers that need the old values of other columns get nulls unless the table is set to REPLICA IDENTITY FULL, which enlarges the WAL.

Ordering is per key, not global. Partitioning by aggregate id preserves each order's history but not the relative order of changes to an order and its customer, and a multi-table transaction arrives as independent events.

The outbox grows forever. Rows must be deleted after capture. Because the router ignores deletes and the log already holds the insert, a common pattern deletes outbox rows soon after writing them, which works only as long as the log is retained until the connector has read it.