You cannot delete the dual write, only move it
How production systems keep a database write and the event that announces it in agreement, and what each remedy costs.
A service commits a change and must announce it, and no mechanism makes both happen or neither. This guide reconstructs where eighteen organisations put that non-atomic seam, from Facebook's Wormhole in 2015 to Zalando's Debezium contribution in December 2025, what it cost them when it broke in three published incidents, and the conditions under which the standard advice is wrong. After reading it an architect can name which of their own second writes is retryable, compute the one capacity number this architecture actually has, and defend the choice in a design review.
Every remedy for the dual write is itself implemented as a dual write, and Kafka's own KIP-618 says so in its motivation: the connector produces records and then writes its offsets, which is the exact gap the outbox was supposed to close.
What you get out of it
- The design variable is not atomicity but the retryability of the second write and the retention of the evidence behind it.
- The position record, meaning the replication slot versus the connector's offset store, is a second authority for one fact; Zalando and Airbyte users independently hit the same divergence, and Debezium's fix was to hard-code the capability off.
- An outbox does not remove the failure, it changes its currency: PagerDuty's held through a six-hour Kafka outage and delivered duplicate webhooks and a backlog instead.
- The only real capacity number here is recoverable downtime, meaning log retention minus current lag, and nobody publishes it as a metric.
- No public postmortem attributes an incident to a committed outbox row that was never published, which is either evidence the pattern works or evidence the failure is undetectable; both readings argue for independent reconciliation. Meanwhile two-phase commit is coming back: KIP-939 was accepted in July 2024, is absent from Kafka 4.0, and its tracking issue was still open in August 2026.
Scope
Why this, now. Kafka is formally reintroducing two-phase commit for exactly this problem while the author of Netflix's own change-capture framework published in August 2026 that change data capture never solved it, so the profession's default answer is being argued about in public for the first time since 2019.
What it does not cover. Sagas and long-running business compensation, multi-master write conflict resolution, analytics replication where staleness is the only concern, and the internals of consensus.
Other field guides
The renewal failed a month before anyone noticed
A field guide to the machine credential lifecycle, built from ten published incidents at Bazel, DigiCert, Microsoft, Let's Encrypt and Ericsson, the …
22 sources · 17 organisations · 4 postmortemsThe retry is the outage
A field guide to metastable failure: the class of outage where the system stays down after the thing that broke it has been repaired, because the ret…
30 sources · 19 organisations · 7 postmortemsThe rule was already written down
A field guide to the machinery between an outage and the next design decision, reconstructed from six first-party incident reports (Cloudflare 2019 a…
34 sources · 24 organisations · 6 postmortems