Orders are occasionally created in the database with no corresponding event reaching the warehouse. No errors are logged. Diagnose and fix.

What is being tested

Recognition of the dual-write problem, which is one of the most common silent correctness bugs in event-driven systems and one of the hardest to attribute after the fact.

The diagnosis

The service does two things that are not one thing:

1. Commits the order to its database.
2. Publishes OrderPlaced to the broker.

These are separate systems, so no transaction spans them. Between step 1 and step 2 the process can crash, the broker can be briefly unavailable, the network can drop, or the pod can be evicted mid-request. The order exists; the event does not; nothing errors, because from the database's point of view the transaction succeeded and from the broker's point of view nothing was ever sent.

That is why there are no logs. The failure is a gap, not an exception.

The mirror-image bug is publishing first: then a rolled-back transaction leaves an event describing an order that does not exist, and the warehouse picks stock for nothing.

The fix: transactional outbox

Insert the event into an outbox table in the same transaction as the order. Now there is one write to one system, and atomicity is restored. A separate relay reads unsent rows, publishes them, and marks them sent.

Two relay implementations:

• Polling — select unsent rows on an interval. Simple and adequate; needs an index on the unsent predicate or it degrades as the table grows.
• Change data capture — tail the database's replication log. No polling load, lower latency, one more piece of infrastructure.

The consequences you must accept

At-least-once delivery. The relay can publish and then crash before marking the row sent. It will republish. Consumers must be idempotent — this is not an optimisation, it is the pattern's stated contract, and skipping it converts lost orders into duplicate shipments.

Outbox lag is a monitored SLI. This is the critical operational insight. A stalled relay produces no errors anywhere: the database is healthy, the consumers are healthy, and the business silently diverges. The only signal is the age of the oldest unsent row, and it must be alerted on.

Pruning. The outbox is a transient buffer. Delete or partition away sent rows, or it becomes the largest table in the database and its index stops fitting in memory.

Per-entity ordering. Publish rows for the same aggregate in insertion order and partition downstream on the aggregate key. Global ordering is neither provided nor usually needed.

What to do about the orders already lost

Reconciliation, and it is worth building permanently rather than as a one-off script: periodically compare orders in the database against events the warehouse acknowledges, within a time window, and re-emit or alert on the differences. Every system with an asynchronous boundary that matters should have this. It catches the outbox bug, the consumer bug you have not found yet, and the manual database edit someone will make during an incident.

Why not two-phase commit

It exists and some brokers support it, but it takes locks across a network boundary, blocks indefinitely if the coordinator dies mid-protocol, and converts several independent availability numbers into their product. The outbox achieves the necessary guarantee with local transactions only, which is why it won.