Transactional Outbox

Definition

The dual-write problem: a service updates its database and publishes an event. These are two systems, so there is no transaction across them. A crash between them leaves the database updated and the event lost — or, if published first, an event describing a change that was then rolled back.

The outbox pattern removes the gap. The event is inserted into an outbox table inside the same transaction as the business change. A separate relay reads the outbox and publishes, marking rows as sent. Atomicity is restored because there is now only one write, to one system.

Why it matters

The failure this prevents is silent and delayed. An order is created; the OrderPlaced event is lost; the warehouse never hears about it. Nothing errors. The customer discovers it days later. These are among the hardest production bugs to attribute, because by the time the symptom appears the log lines are gone.

Implementation patterns

• Relay by polling. A worker selects unsent rows, publishes, marks sent. Simple, adequate, and needs an index on the unsent predicate or the query degrades as the table grows.
• Relay by change data capture. Tail the database's replication log and publish. No polling load, lower latency, and one more piece of infrastructure to run.
• Accept at-least-once. The relay can publish and crash before marking sent. It will republish. Consumers must be idempotent — this is not optional, it is the pattern's stated contract.
• Preserve per-entity ordering by publishing outbox rows for the same aggregate in insertion order, and by partitioning on the aggregate key downstream.
• Prune aggressively. The outbox is a transient buffer; sent rows should be deleted or partitioned away, or it becomes the largest table in the database.

Industry example

Any large marketplace with a strict "the event must reflect what actually committed" requirement converges on this. Uber-scale systems generate state changes — trip started, trip completed, fare adjusted — that feed dozens of downstream consumers including payments, driver earnings, fraud detection and analytics. A lost TripCompleted is a driver who is not paid, which is a support case, a trust problem, and potentially a regulatory one.

What is instructive at that scale is the operational discipline around the relay: outbox lag is a monitored SLI, because a stalled relay produces no errors anywhere — the database looks healthy, the consumers look healthy, and the business is silently diverging. The failure is invisible unless you specifically measure the age of the oldest unsent row.

Failure scenarios

• The relay stalls and nobody notices, because nothing is erroring.
• Consumers assume exactly-once, and the inevitable republish produces double side effects.
• The outbox table grows without pruning until it dominates the database and its index no longer fits in memory.
• Ordering assumed globally rather than per aggregate, so unrelated entities' events appear reordered and someone builds logic on the assumption they will not.

Trade-offs

You gain a genuine atomicity guarantee between state and event, which is otherwise unobtainable without distributed transactions. You pay a write amplification on every transaction, added end-to-end latency (polling interval or CDC lag), a component to operate and monitor, and the permanent obligation of idempotent consumers.

The alternative — publishing first and hoping, or writing first and hoping — is common, cheaper, and wrong in a way that will not show up in testing.

Interview question

"Your service updates a row and publishes an event. Describe every way that can go wrong, and the smallest change that fixes it."