A service writes to its database and then publishes an event to Kafka. Sometimes consumers see an event for a record that does not exist, and sometimes a record exists with no event. Why, and how do you fix it?

What the interviewer is testing

Recognition of the dual-write problem — one of the most common defects in event-driven systems and one that testing rarely catches, because it needs a crash at a specific instant.

Why it happens

Two independent systems, two independent writes, no atomicity between them:

• Database commits, process crashes before publishing → record exists, no event. Downstream systems never learn about it.
• Publish succeeds, database transaction rolls back → event exists, no record. Consumers act on something that does not exist.
• Publish succeeds but the acknowledgement is lost, so the service retries → duplicate event.

Retrying the publish (the fourth option) fixes only the third case and makes nothing else better, because a crashed process is not retrying anything.

The fix: transactional outbox

Write the event into an outbox table in the same database transaction as the business record. Now there is one write, and it is atomic — either both the order and its pending event exist, or neither does.

A separate relay process reads unpublished outbox rows and publishes them to Kafka, marking them sent. Either it tails the database's write-ahead log via change data capture (Debezium is the usual implementation), or it polls the table.

Delivery becomes at-least-once: the relay may publish and crash before marking the row sent, so the event will be republished. Consumers must therefore be idempotent, which they should be regardless.

Why not the other options

Distributed transaction (2PC) across a database and a broker is technically possible with XA but blocks on coordinator failure, holds locks across the network, and is poorly supported by modern brokers. It trades a rare inconsistency for a regular availability risk.

Publish first inverts which failure you get, without removing it, and it is worse: consumers now act on events for records that may never be committed.

Details that separate a good answer

• Ordering. If consumers need per-entity ordering, the relay must preserve outbox order per partition key. CDC gives this naturally; a naive polling relay with concurrency does not.
• Outbox growth. Sent rows need deletion or partitioning, or the table becomes the largest one in the database.
• The listen-to-yourself variant, where the service consumes its own event to update its read model, keeps everything on one path but adds latency before the write is visible.