A service updates its database and then publishes an event. Occasionally downstream systems are missing records. Diagnose and fix.

What the interviewer is testing

Whether you recognise the dual-write problem, which is one of the most common silent inconsistencies in service architectures.

The diagnosis

Two separate systems, no shared transaction. The database commit and the message publish are independent operations, and the process can fail between them.

The database commits, the process crashes or the broker is briefly unavailable, and no event is published. The order exists and nothing downstream knows. There is no error visible to anyone — from the caller's perspective the operation succeeded, which it did.

Reversing the order does not help: publish first, and a subsequent commit failure means an event was published for a change that did not happen.

The frequency matches the failure rate of the window between the two operations, which is why it is occasional and very hard to reproduce.

The fix: transactional outbox

Write the event as a row in an outbox table within the same local transaction as the business data. Both commit or neither does.

A separate process then reads the outbox and publishes — either by polling, or better by tailing the transaction log through change data capture — and marks the row as sent.

The guarantee is at-least-once: a crash after publishing and before marking will republish. So consumers must be idempotent, which is the correct assumption in any case.

The operational details that decide whether it works

Prune the outbox or it becomes the largest table in the database. Monitor publisher lag, because a stalled publisher is completely invisible from the application's point of view — everything looks committed and nothing is flowing. And ensure ordering is preserved if downstream depends on it, which means publishing in outbox insertion order per key.

What a strong answer adds

The alternative worth mentioning: if the event can be derived from the database change itself, change data capture on the business table removes the outbox entirely. That is cleaner where the event maps to a row change, and insufficient where the event carries intent or context the row does not.

Common weak answers

Adding retries around the publish, which narrows the window without closing it. Publishing before committing.