You cannot delete the dual write, only move it

The territory

The problem, stated without naming a pattern: a service commits a change to its own store and must announce that change to systems it does not control, and the two acts cannot be made atomic.

Messages per day read straight from storage transaction logs at Facebook, at least-once delivery, in 2015

Wormhole, NSDI 2015

55%

Of customer logs lost over 3.5 hours when the buffer tier of an event pipeline was overwhelmed by a config error

Cloudflare, Nov 2024

5 orders

Of magnitude slower: a polled outbox query that ran in under a millisecond, then took 18.5 seconds

Trade Republic, Jun 2025

954

Writes present in one datacentre and absent in the other after a 43-second partition, in a single cluster

GitHub, Oct 2018

The canonical framing comes from Confluent's Wade Waldron in May 2024: the dual-write problem occurs when two external systems must be updated in an atomic fashion. The canonical answer arrived five years earlier, on slide 10 of a QCon San Francisco talk by Gunnar Morling, then leading the Debezium project: friends don't let friends do dual writes. Write the intent inside the database transaction, let something else publish it. That advice is correct and it is also incomplete, and the gap between those two facts is what this page is about.

Here is the finding that reorganised the rest of the research. In August 2026 Andreas Andreakis, who built Netflix's DBLog change-capture framework and wrote the paper on it, published a post arguing that change data capture does not solve dual writes. His sentence is worth quoting exactly, because it is the load-bearing claim of this guide:

The remedy for dual writes is itself implemented as a dual write. Andreas Andreakis, August 2026

His argument is mechanical, not philosophical. The pipeline that reads your database log has to do two things it cannot do atomically: deliver the record to the sink, and record how far it got. Apache Kafka's own design record says the same thing from the inside. KIP-618, the proposal that finally brought exactly-once semantics to Kafka Connect source connectors, states in its motivation that the framework periodically writes task offsets to an internal Kafka topic after records reach Kafka, and that without framework-level adaptations, exactly-once delivery for source connectors remains impossible. Chris Egerton, who implemented it, notes that Connect lacked this support for source connectors until version 3.3. Which is to say: for the first six years that the industry told people to stop doing dual writes and use change capture instead, the change-capture framework everybody reached for was itself doing a dual write.

So the useful question is not how to make two writes atomic. It is: where in the system do you want the non-atomic seam to sit, and what does a failure look like when it sits there? Every architecture in this guide answers that question differently, and the answers are legible once you stop looking for the one that eliminates the seam.

Figure 1 · Where the seam is, and the four places teams move it to

Every remedy relocates the crash window rather than closing it. Read the four lower boxes as four different answers to "what is the last thing that can fail, and who notices". Synthesised from Andreakis, 2026, KIP-618 and Confluent Developer.

Diagram source

Scope

This guide covers the seam between one service's own store and the event that announces a change to it: outbox tables, log-based change capture, listen-to-yourself, and the return of two-phase commit. It deliberately does not cover sagas and long-running business compensation, multi-master or cross-region write conflict resolution, analytics replication where staleness is the only concern, or the internals of consensus. Those are adjacent problems with their own literature. The companion field guide in this collection, Retry storms and metastable failure, covers what happens when the backlogs described here are drained badly.

How it is actually built

Seven parts recur across every published system, from Facebook in 2015 to Zalando in 2025. Only three of them are usually drawn, and the one that causes the most incidents is almost never drawn at all.

Put the published architectures side by side and the common shape falls out quickly. Facebook's Wormhole, described at NSDI in 2015, has publishers that directly read the transaction logs maintained by the data storage systems and offers at least once delivery. Airbnb's SpinalTap, open sourced and still readable, is a general-purpose reliable Change Data Capture (CDC) service with sources, destinations and a ZooKeeper state store holding a binlog file name and position. Netflix's DBLog interleaves chunked selects with log events using watermarks so a backfill does not stall the stream. Shopify ran roughly 150 Debezium connectors across 12 Kubernetes pods against more than 100 MySQL shards. Clear Street put an outbox table in front of Debezium. Zalando ran Postgres logical replication into hundreds of event streams. Different decades, different databases, same seven parts.

Figure 2 · The reference architecture, with the part nobody draws

The position record is the divergence point: it is a second, independent statement of the same fact the log already states, and keeping the two in agreement is the dual write you thought you had removed. Reconstructed from Zalando, Shopify, Airbnb SpinalTap and Netflix DBLog.

Diagram source

The intent record, and where it lives

Either you write an explicit row inside the transaction, or you treat the engine's own write-ahead log as the intent record and capture the business tables directly. Shopify took the second route because it could not touch the write path of a monolith sharded across more than 100 databases. Clear Street took the first because it wanted the published contract to differ from the table schema, storing schema_id, partition and an Avro data blob per row.

Runs this way at: Shopify, Clear Street, Zalando

The relay, and what it costs the database

A poller is a query that runs forever against a table with a very high churn rate. A log reader is a replication client that holds the database's log hostage until it catches up. Neither is free, and the bill arrives in a different currency: the poller charges you in query plans and dead tuples, the log reader charges you in disk.

Runs this way at: Trade Republic, Zapier, Airbnb

The position record, and why it diverges

The relay must remember how far it got. Postgres offers a replication slot that remembers on the server. Kafka Connect offers an offset topic that remembers on the client. Run both and you have two authorities for one fact. Zalando's engineers say plainly that they always treated the PostgreSQL replication slot as the authoritative source of truth for stream position, which is exactly why the upstream default broke them.

Runs this way at: Zalando, Airbyte users, Kafka Connect

Two of the seven parts are optional at small scale and become mandatory above it. The first is the backfill path. Netflix built DBLog specifically because the naive answer, lock the table and snapshot it, does not survive contact with a large production database. The paper's contribution is a watermark scheme that lets you interleave transaction log events with rows that we directly select from tables, executing selects in chunks, and it does not use locks and has minimum impact on the source. Shopify hit the same wall from the other side and wrote it down bluntly: Several tables in Shopify's Core monolith are too big to snapshot in any reasonable time-frame. If your only recovery from a broken position record is a full snapshot, then above some table size you have no recovery.

The second is the inbox. Every system here delivers at least once and says so: Wormhole in 2015, SpinalTap by construction, Debezium by design, Clear Street in its own words (you are guaranteed that a corresponding event with the details of what you changed are published to Kafka at-least once). At-least-once is not a caveat in a footnote; it is a requirement placed on every consumer you will ever have, forever. Neither Clear Street nor Shopify publishes how their consumers deduplicate, which is a gap worth noticing. The mechanism is well known, a persisted set of processed event identifiers checked inside the consumer's own transaction, but no source in this hunt publishes a measured duplicate rate. An architect sizing that store is working without numbers.

Figure 3 · The relay's own crash window

This is the seam that change capture inherits rather than removes. Produce first and crash, and the records are delivered twice on restart; commit the position first and crash, and they are never delivered at all. Kafka Connect's fix, shipped in 3.3, is to put both writes in one Kafka transaction. Source: KIP-618 motivation.

Diagram source

The decisions that matter

Six forks, each with a documented choice, a documented rejection, and the condition that flips it. The flip conditions are derived from the stated reasons: where the reason is a constraint, the decision reverses when the constraint does.

Decision: capture the business tables, or write an explicit outbox row?

Chosen

Shopify captured the business tables directly across 100-plus MySQL shards, using Debezium and a Kafka Streams job to fold shard topics into per-table topics keyed by primary key.
It required no change to the monolith's write path, which was the binding constraint.

Rejected

An outbox table, which would have meant touching every write path in a monolith that predates the platform team.
Also rejected: Maxwell's Daemon, SpinalTap and a DBLog-style build, on the grounds that Debezium was by far the most active open-source CDC project.

Flips when

The published contract must differ from the internal table schema. Shopify names this cost itself: breaking schema changes force every downstream consumer to update in lockstep, which couples your internal data model to external systems.
You own the write path and can afford one extra insert per transaction.

Decision: poll the outbox, or read the database log?

Chosen

Clear Street chose log capture over the outbox table, so the publish is driven by the write-ahead log rather than a query.
Trade Republic kept polling and made it survivable by partitioning the outbox on published_at, which took the fetch query from 18.5 seconds back to 1 to 3 milliseconds.

Rejected

Trade Republic rejected the obvious partial index on unpublished rows: under concurrent insert and update it accumulated dead entries and the planner fetched close to 100 million heap rows to find 1,000 unpublished messages.

Flips when

Your database gives you no usable logical replication, or you cannot get an operator to grant replication rights. Then poll, and partition rather than index.
Publish volume is low enough that a poller's constant scan is invisible in the database's load profile. Trade Republic's failure only appears under concurrent insert and mark-published traffic.

Decision: who is authoritative for the stream position, the slot or the offset store?

Chosen

Zalando made the Postgres replication slot authoritative and ran Debezium with the ephemeral MemoryOffsetBackingStore, so there is exactly one record of position.
They ran that configuration for nearly two years on Debezium 2.7.4 and report billions of events with zero detected data loss from this mechanism.

Rejected

The Debezium default, a persistent Kafka Connect offset topic alongside the slot. Debezium's maintainers went further and hard-coded the driver-level keepalive flush to disabled, because it conflicted with Debezium's own LSN management logic.

Flips when

You cannot afford a re-snapshot. A slot-authoritative design has no independent record to fall back on, so losing the slot means backfilling, and Shopify's tables show where that stops being possible.
Multiple independent readers consume one slot, at which point one shared server-side position is wrong for all but one of them.

Figure 4 · Choosing where to put the seam

The first question is the one most teams skip, and it is the only one whose answer cannot be bought with more infrastructure: if the effect is not safe to repeat, no publishing mechanism saves you and the work moves to the receiver. Derived from the decisions above and from Andreakis, 2026.

Diagram source

The fourth decision is the one the industry reversed. In May 2019, Kleppmann, Beresford and Svingen opened their Communications of the ACM article with the sentence that became the profession's default position: Distributed transactions have failed as a mechanism for ensuring consistency across heterogeneous storage technologies in today's large-scale applications. Clear Street's 2020 post repeats the received wisdom without argument, rejecting two-phase commit as slow and, frankly, complicated to implement.

Then Kafka started building it. KIP-939, Support Participation in 2PC, was accepted in July 2024 and states the motivation in language that could have come from any of the blog posts above: Dual write is easy to implement but means that whenever there is a failure there is a high likelihood that the log and the database will diverge. It adds an enable2Pc flag to InitProducerId, a prepareTransaction() call returning a serialisable state, and a transaction.two.phase.commit.enable producer setting. Artem Livshits presented it at Current 2024 as a recipe guaranteeing that events are committed to Kafka iff changes are committed to the database.

Two things about that revival deserve an architect's attention. First, the KIP writes its own costs down honestly: the transaction is never aborted automatically, and consumers reading at read_committed cannot consume past the ongoing transaction. A coordinator that dies mid-prepare stalls every reader of that partition until an operator runs forceTerminateTransaction(). That is the classic blocking property of two-phase commit, restated in Kafka's vocabulary, and it is the reason the pattern was abandoned in the first place. Second, it has not shipped. Kafka 4.0 landed on 18 March 2025 without it. As of 13 August 2026 the tracking issue KAFKA-15370 is still open and unresolved with a fix version of 4.5.0, three years after it was filed. Anyone planning around 2PC-with-Kafka today is planning around a design document.

There is a fifth decision hiding inside Flink, and it is the sharpest evidence in this guide that the seam is real rather than theoretical. FLIP-319 explains why Flink wants KIP-939: the current exactly-once Kafka sink relies heavily on Java reflection in order to bypass the Kafka transaction protocol, and data loss can occur when Kafka aborts a successfully checkpointed transaction due to timeout. The most widely deployed exactly-once sink in stream processing achieves its guarantee by reaching around the protocol, and has a documented loss mode. That has been written down since August 2023.

Six decisions, the stated reason, and the condition that reverses it.
Decision	Chosen by	Rejected	Because	Flips when	Evidence
Where the intent record lives	Business tables (Shopify)	Outbox table	Could not change a sharded monolith's write path	The published contract must differ from the table schema	Shopify, 2021
How the relay reads	Log capture (Clear Street)	Polling	Publish is driven by the commit, not a query	No logical replication available, or publish volume is low	Clear Street, 2020
How the outbox is pruned	Partition and truncate (Trade Republic)	Partial index plus delete	Dead index entries forced ~96.6M heap fetches for 1,000 rows	Never, on Postgres, above trivial volume	Trade Republic, 2025
Authority for stream position	Replication slot (Zalando)	Persistent offset topic	One record of position cannot disagree with itself	Re-snapshot is unaffordable, or many readers share one slot	Zalando, 2025
Delivery guarantee	At-least-once (everyone)	Exactly-once machinery	Cheaper, and the consumer has to be idempotent anyway	The effect is a payment or an email, where repeating is the failure	Wormhole, 2015
Coordination model	Log-first, no 2PC (2019 consensus)	Two-phase commit	Distributed transactions have failed	KIP-939 ships and you can operate a coordinator, accepting blocked read_committed consumers	KIP-939, 2024

One decision remains genuinely open, and the open-ness is itself the finding. In July 2026 a contributor opened a discussion-only pull request against Debezium, DDD-53, proposing a trigger-less outbox poller that detects row changes without a replication slot at all, by diffing an in-memory baseline at a stated cost of roughly 15 MB per million watched rows. The only maintainer response was procedural (This DDD was moved to debezium/debezium-design-documents#51) and it was closed on 14 August 2026 with no technical objection recorded. Read that as an honest signal: the design space between "hold a replication slot" and "run a query forever" has no settled answer, and the project's own contributors are still probing it.

What broke in production

Five failure classes with published evidence, drawn from three postmortems, two issue threads and four engineering accounts, plus a sixth class that nobody has written up. The absence is the interesting part.

Class A · The evidence expires

Eng blog

The log is evidence with a retention policy

AssumptionThat a change-capture reader either works or alerts, so a stopped connector is a paging problem rather than a data problem.

What happenedGunnar Morling's answer to his own question is that Debezium by itself should never miss any event, but that due to operational deficiencies portions of the database's transaction log get discarded before Debezium gets a chance to capture them. MySQL's binlog_expire_logs_seconds defaults to 2,592,000 seconds, so the window is 30 days and then the evidence is gone.

Blast radiusSilent. There is no error at the moment of loss, only a gap discovered later by reconciliation, if anyone reconciles.

FixAlert on connector liveness against retention, not on connector errors.

Design ruleYour recoverable outage length is log retention minus current lag. Publish that number as a service level objective and alarm on it, because it is the only number that tells you how long you may be down before the loss becomes permanent.

SourceGunnar Morling, 14 November 2023

Source

A sync that succeeds and captures nothing

AssumptionThat a connector reporting success has read the log.

What happenedAn Airbyte user syncing a 900 GB Postgres table with roughly 50 GB of write-ahead log per day saw runs complete with zero records and the log line WAL resume position 'null' discovered. Raising the initial wait from 300 to 1,200 seconds improved matters without fixing them.

Blast radiusSporadic and undetectable from the pipeline's own status: the run is green.

FixNone recorded in the thread; it was closed without a documented resolution.

Design ruleNever treat "job succeeded" as evidence of capture. Emit a heartbeat row through the same path as real changes and alarm on its absence, which converts a silent gap into a loud one.

SourceAirbyte issue 31312, 2023

Class B · The position record diverges from the log

Case study

The fix that worked for one team was disabled for everyone

AssumptionThat the replication slot and the connector's offset store are two views of one truth.

What happenedZalando's single biggest operational issue was that on low-activity databases replication slots wouldn't advance without table activity, causing WAL to pile up until disk space ran out. Their fix made the Postgres JDBC driver answer keepalives and flush the position. Debezium's maintainers later hard-coded that same driver feature off, with withAutomaticFlush(false), because for users with a persistent offset topic the slot would advance ahead of the stored offset.

Blast radiusZalando could not upgrade Debezium without reintroducing an unbounded disk-growth failure across 100-plus Kubernetes clusters.

FixZalando contributed lsn.flush.mode and offset.mismatch.strategy upstream, released in Debezium 3.4.0.Final, turning a hard-coded assumption into a configuration decision.

Design ruleIf two components can both record the same position, name one of them authoritative in writing, and check that your tooling's defaults agree with you. A framework default is an architectural decision somebody else made about your system.

SourceZalando Engineering, 19 December 2025

Source

An error string that names the divergence

AssumptionThat resetting a pipeline from the console clears its state.

What happenedAirbyte users hit Saved offset is before replication slot's confirmed lsn. A full clear preserved stale shared Debezium state whenever orphaned per-stream state existed, leaving the connection permanently unrecoverable through the user interface and requiring direct state API calls.

Blast radiusSyncs fail indefinitely. Reported 20 July 2026 and still open with no assignee.

FixProposed by the reporter, not yet by maintainers: treat stream state absent from the current catalogue as resettable.

Design ruleBuild the reset path before you need it, and test it on a pipeline whose catalogue has changed since it was created. Two organisations independently hit the same divergence, which makes it a property of the architecture rather than of one tool.

SourceAirbyte issue 82266, July 2026

Figure 5 · How the two records of one position drift apart

The driver advanced the server-side slot on a keepalive while the connector's own offset store stayed behind, so the restart asked for a position the server had already discarded. Reconstructed from Zalando's account and the error text in Airbyte issue 82266.

Diagram source

Class C · The buffer becomes the failure

Postmortem

55% of logs lost when a failsafe opened

AssumptionThat failing open is the safe direction, and that a buffering tier provisioned for one million buffers has headroom.

What happenedA misconfiguration produced a blank Logfwdr config meaning no customer wanted logs. The revert, five minutes later, triggered a latent failsafe that forwarded events for all customers. Buftee went from 40 million buffers globally to roughly forty times that, and became so overloaded that we could not interact with them normally. A full reset and restart was required.

Blast radiusAbout 55% of the roughly 4.5 trillion event logs pushed to customers daily were lost over about 3.5 hours on 14 November 2024.

FixAlerts on misconfiguration, and regular overload tests, having neglected to regularly test that the broader system was capable of handling a fail open event.

Design ruleIn an at-least-once pipeline the buffer tier is where a configuration error turns into permanent loss, and a failsafe you have never exercised at full load is a second untested system in the delivery path. Test the fail-open state, not only the fail-closed one.

SourceCloudflare, 26 November 2024

Eng blog

The safety mechanism that fills the disk

AssumptionThat a replication slot is purely protective, since it stops the database discarding log the reader has not seen.

What happenedThe slot does its job whether or not anything is reading. Morling's summary is that Postgres slots prevent loss by default but risk disk exhaustion, and that max_slot_wal_keep_size, added in Postgres 13, reintroduces the loss risk once you configure it. Zalando hit the same mechanism from the idle-database direction.

Blast radiusThe primary database stops accepting writes. The failure lands on the transactional system, not on the pipeline that caused it.

FixBound the slot and accept bounded loss, or leave it unbounded and accept the disk risk. There is no third option.

Design ruleSetting a cap on retained log is a decision to prefer availability of the source database over completeness of the event stream. Write that down as a decision record, because whoever sets that parameter at 3am is making an architectural choice.

SourceGunnar Morling, 2023

Class D · At-least-once reaches the customer

Postmortem

The outbox held, and customers got duplicate webhooks

AssumptionThat a durable outbox turns a broker outage into a non-event.

What happenedA new API usage tracking feature instantiated a new Kafka producer for every API request, generating nearly 4.2 million extra producers per hour at peak. This is 84 times higher than our typical number of new producers, exhausting JVM heap. PagerDuty's outbox did exactly what it was built to do: No previously accepted events or data were lost during or after the incident. What customers experienced instead was delay and repetition, since affected customers may have received duplicate webhooks.

Blast radiusTwo incidents on 28 August 2025, 03:53 to 10:10 UTC and 16:38 to 20:24 UTC. At peak 18.87% of create requests returned 502, about 23% of customers saw notifications delayed by more than five minutes, and chat integrations were impacted for 515 minutes.

FixJVM and producer-level monitoring, stricter change management with slower ramp-up windows, and monthly chaos exercises on the incident workflow itself.

Design ruleThe outbox does not remove the failure, it changes its currency from lost data to latency and duplicates. Both of those are visible to your customers, so the receiving contract has to say what a duplicate means before you adopt the pattern, not after your first incident.

SourcePagerDuty Engineering, 2025

Postmortem

Two stores, each holding writes the other lacks

AssumptionThat an automated failover leaves one authoritative copy.

What happenedA 43-second network partition during optics replacement let Orchestrator promote west-coast primaries while east-coast primaries had already accepted writes. GitHub's own words: Because the database clusters in both data centers now contained writes that were not present in the other data center, we were unable to fail the primary back over. One busy cluster had 954 writes in the affected window.

Blast radius24 hours and 11 minutes of degraded service from 21 October 2018, with over five million hook events and 80 thousand Pages builds queued by the end of it.

FixOrchestrator was reconfigured to prevent promotion across regional boundaries, and the failure-injection practice was formalised.

Design ruleDivergence is cheap to create and expensive to resolve: seconds of partition bought a day of manual reconciliation. Build the diff-and-repair tool before you need it, and note that the backlog outlives the incident by hours, which is where the retry storm starts.

SourceGitHub, 30 October 2018

Class E · The outbox itself becomes the bottleneck

Eng blog

A fetch query that went from under a millisecond to 18.5 seconds

AssumptionThat a partial index on unpublished rows keeps the outbox fetch cheap.

What happenedUnder concurrent insert and mark-published traffic, dead index entries forced visibility checks against the heap. Trade Republic measured a plan that fetches close to 100 million rows from the heap, which takes over 18.5 seconds to execute. This means a slowdown 5 orders of magnitude, to find 1,000 unpublished messages.

Blast radiusPublish latency collapses precisely when publish volume is highest, which is the definition of a load-correlated failure.

FixPartition the outbox on published_at so unpublished rows live in their own small partition. Result: a consistent result of 1000 heap fetches and an execution time of 1-3 ms.

Design ruleAn outbox is a queue implemented in a table optimised for something else. On Postgres, partition it and truncate the published side; do not index-and-delete. This failure does not appear in any single-threaded benchmark, which is why so many teams meet it in production.

SourceSadeq Dousti, Trade Republic, 8 June 2025

Eng blog

The outbox that got moved off the write path

AssumptionThat a local durable store on each node is the cheapest way to keep accepting events while the broker is unavailable.

What happenedZapier ran an outbox as 50 sharded SQLite files per pod in write-ahead-log mode on EBS, behind a StatefulSet, peaking at fifteen thousand events per second. They hit SQLITE_BUSY under load, volume growth needing aggressive vacuum tuning, slow pod start because of vacuum on large files, and slow recovery after a backlog.

Blast radiusOperational rather than customer-facing: deployment inflexibility and slow scaling response, which is what makes the next incident worse.

FixA sidecar built on object storage and a managed queue, removing the local write from the hot emit path.

Design ruleDurability on the write path buys availability and charges you in storage operations and stateful deployment. If your bus already has a durable, replayable backlog, a second durable store in front of it may be paying twice for one guarantee.

SourceZapier Engineering, 2026

The missing failure class

Class F would be: an outbox row committed inside the transaction and then never published, causing a customer-visible divergence. That is the failure the whole pattern exists to prevent, and no public postmortem in this hunt attributes an incident to it. Every published incident is about the transport, the buffer, the position record, or the database underneath. Two readings are available and they point the same way for a designer: either the pattern genuinely works, or the failure is so silent that nobody detects it well enough to write it up. Both readings argue for the same control, which is an independent reconciliation job that compares the source of truth against the sink and reports drift as a metric rather than as an alert nobody owns.

Numbers you can plan against

Everything quantitative found in the hunt, with the organisation, the context it was measured in, and the date. Where a number is a vendor claim or a derivation rather than a measurement, it says so.

Figure 6 · The four states of a capture pipeline, and the only irreversible one

Lag is not a performance metric here, it is a countdown. The transition an operator must never reach is the one where lag exceeds retention, because the only exit is a backfill whose cost grows with table size. Synthesised from Morling, 2023, Zalando, 2025 and DBLog, 2020.

Diagram source

Measured figures from primary sources. All checked 29 August 2026.
Metric	Value	At	Context	As of	Source
Capture throughput, sustained	~65,000 rec/s	Shopify	Black Friday and Cyber Monday, 100-plus MySQL shards	2020	Shopify
Capture throughput, peak	100,000 rec/s	Shopify	Same event, spike handling	2020	Shopify
Write-to-bus latency	p99 < 10 s	Shopify	MySQL insert to availability in Kafka	2021	Shopify
Connector fleet	~150 / 12 pods	Shopify	Debezium connectors across Kubernetes pods	2021	Shopify
Captured data retained	400 TB+	Shopify	Compacted CDC topics in one Kafka cluster	2021	Shopify
Capture throughput	100k+ ev/s	Zalando	Combined connectors across 100-plus Kubernetes clusters	2025	Zalando
Log-tailing pub-sub, steady state	35 GB/s	Facebook	Wormhole reading MySQL, HDFS and RocksDB logs	2015	NSDI '15
Same system, recovery burst	200 GB/s	Facebook	Replay after failure; 5 trillion messages per day	2015	NSDI '15
Outbox emit rate	15,000 ev/s	Zapier	Peak, 50 SQLite shards per pod on EBS	2026	Zapier
Polled outbox, degraded plan	18,553 ms	Trade Republic	96,633,220 heap fetches to find 1,000 unpublished rows	2025	Trade Republic
Same query, partitioned outbox	1–3 ms	Trade Republic	1,000 heap fetches, consistent across runs	2025	Trade Republic
Event delivery pipeline	4.5 trillion/day	Cloudflare	Logs pushed to customers; 45 PB/day uncompressed upstream	2024	Cloudflare
Loss during buffer overload	55% / 3.5 h	Cloudflare	Buffer count grew roughly 40x from 40 million	2024	Cloudflare
Divergent writes after a 43 s partition	954	GitHub	One cluster; recovery took 24 h 11 min in total	2018	GitHub
Event backlog after the incident	5,000,000+	GitHub	Hook events queued, plus 80,000 Pages builds	2018	GitHub
Producer leak at peak	4.2M/hour	PagerDuty	84x normal new-producer rate; heap exhaustion followed	2025	PagerDuty
Default MySQL log retention	2,592,000 s	MySQL	binlog_expire_logs_seconds, so 30 days of recoverable downtime	2023	Morling
Log volume on one busy table	~50 GB/day	Airbyte user	900 GB Postgres table under logical replication	2023	Issue 31312
Time from KIP acceptance to shipping	> 2 years	Apache Kafka	KIP-939 accepted July 2024; KAFKA-15370 still open, fix version 4.5.0	2026	KAFKA-15370
Exactly-once for source connectors	Kafka 3.3	Kafka Connect	The first release in which a source connector could be exactly-once	2023	Egerton

Read these carefully

Measured: every row above comes from a primary account by the team that ran the system, except the Wormhole figures, which come from a peer-reviewed paper, and the Airbyte row, which is a user report rather than a controlled measurement. Claimed but unverified: Zalando's "zero detected data loss over nearly two years" is a detection claim, not a loss claim; nothing in the post describes the reconciliation that would license the stronger reading. Derived: the useful planning quantity is not on the table, because nobody publishes it. It is recoverable downtime = log retention minus current lag. On MySQL defaults that starts at 30 days and shrinks with every hour of lag; on a Postgres slot it is unbounded until the disk fills, and on a capped slot it is whatever max_slot_wal_keep_size divided by your log generation rate works out to. Compute it, publish it on a dashboard, and alarm at half. Unknown: the duplicate rate under normal operation and after a rebalance. Not one of the thirty-five sources in this hunt publishes it, which means every inbox table in production is sized by guess.

Two shape observations worth carrying into a capacity conversation. First, the recovery burst is the number that sizes the system, not the steady state: Facebook's Wormhole moves 35 GB/s normally and up to 200 GB/s during replay, a factor of nearly six. If your bus, your consumers and your database can only absorb the steady state, then every recovery is a second outage, which is what GitHub's five million queued hook events describe from the other end. Second, latency in these systems is bimodal, not distributed. Shopify's p99 under 10 seconds and Trade Republic's 1 to 3 milliseconds are both healthy-state numbers. The unhealthy state is not the ninety-ninth percentile of the same distribution, it is a different regime measured in hours. Alerting on a latency percentile will therefore miss the failure that matters; alert on lag against retention instead.

The evidence wall

Every source behind this page, graded by what it can prove. Filter by kind. The full ledger, with one row per claim and the supporting quote copied verbatim, ships beside this file as sources.md.

Postmortem PagerDuty2025-09

August 28 Kafka Outages: What Happened and How We're Improving

Two incidents in one day caused by a producer instantiated per API request. The transactional outbox prevented loss and converted the failure into a backlog plus duplicate webhooks. Rare in publishing both the mechanism and the customer-visible consequence of at-least-once delivery.

Carry forwardDurability changes the currency of the failure from lost data to delay and duplicates, and both are visible to your customers.

pagerduty.com/eng

Postmortem Cloudflare2024-11

Cloudflare incident on November 14, 2024, resulting in lost logs

A blank configuration, a five-minute revert, and an untested fail-open path that multiplied buffer count roughly fortyfold. The clearest public account of how the buffering tier of an event pipeline becomes the loss mechanism.

Carry forwardLoad-test the fail-open state of every delivery component, because a failsafe never exercised at full scale is an untested system in the critical path.

blog.cloudflare.com

Postmortem GitHub2018-10

October 21 post-incident analysis

Forty-three seconds of partition produced writes in two datacentres that could not be reconciled automatically, and a day of degraded service. The definitive worked example of what divergence costs once it exists.

Carry forwardWrite the diff-and-repair tool before you need it; the backlog outlives the incident and is where the next failure begins.

github.blog

Eng blog Andreas Andreakis2026-08

Change-Data-Capture Doesn't Solve Dual-Writes

The author of Netflix's DBLog framework arguing that the remedy contains the disease. Distinguishes retryable sink writes from non-retryable side effects, and points out that replayable history has an expiry date.

Carry forwardAsk of every design: which of my second writes is retryable, and for how long is the evidence behind it retained.

aandreakis.com

Decision record Apache Kafka2023-06

KIP-618: Exactly-Once Support for Source Connectors

Kafka's own admission that a source connector produces records and then writes its offsets, and that the gap is unavoidable without framework changes. Includes ten rejected alternatives, among them per-partition producers and record deduplication.

Carry forwardThe relay is a dual writer too. If your connector predates Kafka 3.3, its guarantee is at-least-once regardless of what the diagram says.

cwiki.apache.org

Decision record Apache Kafka2024-07

KIP-939: Support Participation in 2PC

Two-phase commit returning to the stack that was built to avoid it, with the costs stated openly: transactions never abort on their own, and read_committed consumers block behind a prepared transaction until an operator intervenes.

Carry forwardAccepted is not shipped. Treat 2PC-with-Kafka as a design document until KAFKA-15370 closes.

cwiki.apache.org

Decision record Apache Flink2023-08

FLIP-319: Integrate with Kafka's Support for Proper 2PC Participation

Flink's exactly-once Kafka sink bypasses the transaction protocol using Java reflection and can lose data when Kafka aborts a checkpointed transaction on timeout. Written by the maintainers, about their own code.

Carry forwardAn exactly-once label on a connector is a claim about a configuration, not a property of the system. Read the design doc before believing it.

cwiki.apache.org

Case study Zalando2025-12

Contributing to Debezium: Fixing Logical Replication at Scale

Seven years of Postgres logical replication at scale, the WAL-growth failure on idle databases, the driver fix, and the upstream decision to hard-code that fix off. Ends with two new configuration options contributed back.

Carry forwardName one authority for stream position in writing, then check your framework's default agrees with you.

engineering.zalando.com

Case study Shopify2021-03

Capturing Every Change From Shopify's Sharded Monolith

Log capture across more than 100 MySQL shards with real figures, plus the two honest complaints: tables too large to snapshot, and breaking schema changes that couple every downstream consumer to the internal data model.

Carry forwardCapturing business tables is free at the write path and expensive at the contract. Budget for a schema boundary you did not create.

shopify.engineering

Eng blog Trade Republic2025-06

PostgreSQL + Outbox Pattern Revamped, Part 1

Query plans, heap fetch counts and execution times for a polled outbox before and after partitioning. The most quantitatively honest outbox post found in this hunt.

Carry forwardPartition the outbox on published_at and truncate; a partial index plus delete degrades by five orders of magnitude under concurrency.

dev.to

Eng blog Zapier2026

Lessons from using the outbox pattern at scale

A local sharded SQLite outbox on the emit path at fifteen thousand events per second, the operational drag it produced, and the object-storage sidecar that replaced it. Rare in describing a pattern being retired rather than adopted.

Carry forwardIf the bus already gives you a durable replayable backlog, a second durable store in front of it may be paying twice for one guarantee.

zapier.com/blog

Eng blog Gunnar Morling2023-11

Can Debezium Lose Events?

The former Debezium lead answering the question directly: the tool does not lose events, operations do, when the log is discarded before the reader catches up. Names the exact parameters on MySQL and Postgres that set the boundary.

Carry forwardRecoverable downtime equals retention minus lag. It is the only capacity number this architecture really has.

morling.dev

Eng blog SQUERundated

Stop overusing the outbox pattern

The dissent. David Leitner argues the outbox turns the database into the bottleneck of an architecture adopted to avoid one, and proposes listen-to-yourself and event sourcing as simpler answers for some cases. No page date, so treat as current opinion rather than a dated position.

Carry forwardAsk whether the pattern is protecting a guarantee you actually need, or is arriving by default in a code review.

squer.io

Eng blog Clear Street2020-10

Designing Clear Street's First Transactional Outbox Pattern

An outbox table read by Debezium in a brokerage, with the schema of the event table spelled out and 2PC explicitly rejected. Promises a follow-up on what went wrong that was never published.

Carry forwardCarry the Kafka partition key and the schema registry id as columns in the outbox row, so routing is data rather than code.

clearstreet.io

Source Airbyte2026-07

Issue 82266: full reset never erases global CDC shared state

The offset-versus-slot divergence with an error string attached, plus a reset path that cannot recover from it through the interface. Open, unassigned, with a workaround supplied by the reporter.

Carry forwardTest your reset path on a pipeline whose stream catalogue has changed since creation. That is where the state machine breaks.

github.com/airbytehq

Source Airbyte2023-10

Issue 31312: no records synced via CDC for large WAL

A 900 GB table, roughly 50 GB of log per day, and syncs that report success while capturing nothing after a null resume position. Closed without a documented fix.

Carry forwardSend a heartbeat change through the real capture path and alarm on its absence; a green job is not evidence of capture.

github.com/airbytehq

Source Debezium2026-08

PR 7687: DDD-53, a trigger-less outbox polling connector

A discussion-only proposal to detect row changes without a replication slot, closed unmerged after a purely procedural response redirecting it to the design-documents repository. No technical objection recorded.

Carry forwardThe middle ground between holding a slot and polling forever is unsettled. If neither fits your constraints, you are not doing it wrong.

github.com/debezium

Source Airbnbrepo

SpinalTap

An independently built change-capture service, open sourced, with a source layer, a Kafka destination and a ZooKeeper state store holding binlog file, position and next position. Useful as convergent evidence for the reference architecture.

Carry forwardTwo teams who never spoke to each other both externalised the position record. That is a component, not an implementation detail.

github.com/airbnb

Source Apache Kafka2026-08

KAFKA-15370: Support Participation in 2PC

The tracking issue for KIP-939. Created August 2023, last updated August 2026, status open, resolution unresolved, fix version 4.5.0. Sub-tasks partially complete.

Carry forwardCheck the JIRA, not the KIP status, before putting a Kafka feature on a roadmap.

issues.apache.org

Paper Netflix2020-10

DBLog: A Watermark Based Change-Data-Capture Framework

Andreakis and Papapanagiotou on interleaving chunked selects with log events using watermarks, without locks, so a backfill does not stall the stream. In production at Netflix across tens of microservices at the time of writing.

Carry forwardBackfill is a first-class component, not an emergency procedure. If your only snapshot mode locks the table, you have no recovery above a certain size.

arxiv.org/abs/2010.12597

Paper CACM 62(5)2019-05

Online Event Processing: Achieving consistency where distributed transactions have failed

Kleppmann, Beresford and Svingen make the case that append-only logs can provide atomicity and invariant enforcement without distributed transactions. The intellectual foundation of the log-first default, and the sentence everybody quotes.

Carry forwardThe log-first argument is about ordering, not about atomicity of side effects. Read it before citing it in a review.

martin.kleppmann.com

Paper Facebook, NSDI '152015

Wormhole: Reliable Pub-Sub to Support Geo-replicated Internet Services

Publishers read the storage systems' own transaction logs, delivery is at-least-once, and the deployment moved 35 GB/s steady with bursts to 200 GB/s. Proof that the shape in figure 2 is a decade old.

Carry forwardSize the pipeline for the recovery burst, not the steady state. The published ratio here is close to six to one.

blog.acolyer.org

Talk QCon SF2019-11

Practical Change Data Streaming Use Cases With Apache Kafka and Debezium

Gunnar Morling's 65-slide deck. Slide 10 and slide 60 carry the line that became the industry default position, and slides 41 and 42 give the outbox table design that most implementations still follow.

Carry forwardThe slogan is sound and incomplete. It tells you to move the seam; it does not tell you where the seam lands.

speakerdeck.com

Talk Current 20242024

Atomic Dual-write Recipes with Kafka Two Phase Commit (KIP-939)

Artem Livshits of Confluent presenting 2PC as the recipe that makes events commit to Kafka if and only if changes commit to the database. Describes intent; the feature had not shipped at the time of the talk or at the time of writing.

Carry forwardWhen a vendor session describes a guarantee, check which release it lands in before it enters your design.

current.confluent.io

Talk Kafka Summit London2023

Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors

Chris Egerton, who implemented KIP-618, on why Connect lacked this support for source connectors until version 3.3 and what it took to add it. Dates the gap precisely.

Carry forwardIf your Connect deployment is older than 3.3, or your connector does not opt in, the pipeline is at-least-once end to end.

confluent.io

Vendor Confluent2024-05

Understanding the Dual-Write Problem and Its Solutions

Wade Waldron's statement of the problem and the four-option menu: outbox, event sourcing, listen-to-yourself, and the 2PC family. Useful as the canonical framing; light on failure modes, as vendor material tends to be.

Carry forwardThe outbox is restricted to transactional databases, which quietly rules it out for a large part of a modern estate.

confluent.io/blog

Vendor Confluent Developercourse

Designing Event-Driven Microservices: The Listen to Yourself Pattern

Publish first, then consume your own event to update your database. Removes the dual write and states the price plainly: a caller that reads immediately after writing won't find what it is looking for, and post-response validation failures create inconsistencies the caller never learns about.

Carry forwardThis is the cheapest answer available, and it costs read-your-writes. That is an API contract decision, not an infrastructure one.

developer.confluent.io

Vendor Apache Kafka2025-03

Apache Kafka 4.0.0 Release Announcement

Released 18 March 2025. The headline proposals are 848, 932, 966, 996, 890, 1102 and 653. KIP-939 is not among them, which is the cleanest way to date the gap between the 2PC design and its availability.

Carry forwardRelease notes are the cheapest fact-check available for a roadmap claim about an open-source feature.

kafka.apache.org

Build a miniature, then productionise it

Seven rungs. The first three are an evening each and produce a toy. Rung four is where the toy becomes an operational system, and rungs five to seven are the ones that would have prevented the incidents in section 04.

Reproduce the failure the pattern exists to prevent

One service, one Postgres table, one broker. Commit the row, then publish. Kill the process between the two with a hard signal, in a loop, under concurrent load. Count the rows with no corresponding message.

Done when: you have a non-zero divergence count and can state it as a rate per thousand requests. Teaches: the seam is not theoretical, and its width is a function of how long your publish call takes.

Add an outbox and prove the divergence goes to zero

Insert the event row in the same transaction as the business change. Add a poller that selects unpublished rows, publishes, and marks them. Repeat the kill loop.

Done when: divergence is zero and duplicate count is greater than zero. Teaches: you did not remove the problem, you exchanged missing messages for repeated ones.

Break the poller the way Trade Republic did

Run inserts and mark-published updates concurrently for long enough to accumulate dead tuples, with a partial index on unpublished rows. Watch the plan with EXPLAIN (ANALYZE, BUFFERS) until heap fetches explode. Then partition on published_at and measure again.

Done when: you have both query plans side by side and can explain the visibility check that causes the difference. Teaches: an outbox is a queue built in a table designed for something else, and the failure is invisible to a single-threaded benchmark.

Swap the poller for log capture, then abuse the slot

Point Debezium at the outbox table through a replication slot. Then stop the connector and keep writing. Watch pg_replication_slots and the disk. Set max_slot_wal_keep_size and repeat, and observe that the failure changes from disk exhaustion to permanent event loss.

Done when: you have caused both failure modes deliberately and can say which one your production environment is currently configured for. Teaches: that parameter is an architectural decision about whether the source database or the event stream is more important.

Make the consumer idempotent and measure the duplicate rate

Add an inbox table keyed by event id, checked and inserted inside the consumer's own transaction. Force consumer restarts and rebalances under load and count how many duplicates the inbox absorbs.

Done when: you have a duplicate rate for your own system, since no public source publishes one. Teaches: how large the inbox has to be and how long entries must be retained, which is the retention decision nobody documents.

Instrument recoverable downtime, not latency

Emit two gauges: current lag in log bytes or time, and configured retention. Publish their difference as a single number on a dashboard. Alarm at half, page at a quarter. Add a heartbeat write that flows through the real capture path so a silent stall raises an alert instead of a green run.

Done when: stopping the connector triggers the alarm before any data becomes unrecoverable. Teaches: the operational metric of this architecture is a countdown, not a percentile.

Build reconciliation, and treat drift as a metric

A scheduled job that compares a checksum of the source of truth against the sink over a rolling window and emits the difference. Then exercise the repair path: drop the position record and re-snapshot with chunked selects, DBLog-style, while the stream keeps flowing.

Done when: you can recover from a lost slot without stopping writes, and drift appears on a graph rather than in a support ticket. Teaches: the control that covers the failure class nobody has published a postmortem about.

Keep hunting

The queries that actually produced the material above, grouped by what they surface. The page ages; the method does not.

Production experience, not tutorials

"transactional outbox" production "we" lessons learned engineering blog
"dual write" "change data capture" "we replaced" OR "we moved off"
"outbox" polling postgres "we killed" OR "we removed" bloat vacuum
intitle:"how we" change data capture at scale -tutorial

Failures, in the vocabulary practitioners use

postgres "replication slot" postmortem "disk full" WAL CDC outage
incident "duplicate events" OR "missing events" connector CDC "root cause"
"saved offset is before replication slot" site:github.com
"lost logs" OR "lost events" incident report pipeline buffer overload

The argument, not the conclusion

KIP "two phase commit" producer external transaction rejected alternatives
repo:debezium/debezium is:pr is:closed is:unmerged outbox
"exactly-once" source connectors "rejected alternatives" site:cwiki.apache.org
FLIP OR KIP "data loss can occur" transaction timeout checkpoint

The chain from paper to production

watermark based change data capture framework arxiv
pub-sub "transaction logs" geo-replicated NSDI paper
"online event processing" distributed transactions have failed
<paper author name> blog "dual writes" OR "change data capture"

The single highest-yield move in this hunt was author chaining. Searching the name of the DBLog paper's first author produced a personal blog post, published six years after the paper and three weeks before this guide was written, in which he argues against the received reading of his own work. That post reframed the whole page. Papers tell you what somebody built; the author's blog five years later tells you what they think of it now.

References

PagerDuty, August 28 Kafka Outages: What Happened and How We're Improving PagerDuty Engineering, 2025. Checked 2026-08-29.
Cloudflare, Cloudflare incident on November 14, 2024, resulting in lost logs Cloudflare blog, 26 November 2024. Checked 2026-08-29.
GitHub, October 21 post-incident analysis GitHub blog, 30 October 2018. Checked 2026-08-29.
Andreas Andreakis, Change-Data-Capture Doesn't Solve Dual-Writes Personal blog, 5 August 2026. Checked 2026-08-29.
Apache Kafka, KIP-618: Exactly-Once Support for Source Connectors Apache Software Foundation, accepted 2021, last updated 7 June 2023. Checked 2026-08-29.
Apache Kafka, KIP-939: Support Participation in 2PC Apache Software Foundation, accepted, last updated 23 July 2024. Checked 2026-08-29.
Apache Kafka, KAFKA-15370: Support Participation in 2PC (KIP-939) ASF JIRA, created 17 August 2023, updated 13 August 2026. Checked 2026-08-29.
Apache Flink, FLIP-319: Integrate with Kafka's Support for Proper 2PC Participation (KIP-939) Apache Software Foundation, last updated 18 August 2023. Checked 2026-08-29.
Apache Kafka 4.0.0 Release Announcement Apache Software Foundation, 18 March 2025. Checked 2026-08-29.
Conor Gallagher, Contributing to Debezium: Fixing Logical Replication at Scale Zalando Engineering, 19 December 2025. Checked 2026-08-29.
John Martin and Adam Bellemare, Capturing Every Change From Shopify's Sharded Monolith Shopify Engineering, 12 March 2021. Checked 2026-08-29.
Sadeq Dousti, PostgreSQL + Outbox Pattern Revamped, Part 1 Trade Republic Engineering, 8 June 2025. Checked 2026-08-29.
Zapier, Lessons from using the outbox pattern at scale Zapier Engineering, 2026. Checked 2026-08-29.
Gunnar Morling, Can Debezium Lose Events? morling.dev, 14 November 2023. Checked 2026-08-29.
David Leitner, Stop overusing the outbox pattern SQUER, undated. Checked 2026-08-29.
Clear Street, Designing Clear Street's First Transactional Outbox Pattern Clear Street, 1 October 2020. Checked 2026-08-29.
Airbyte issue 82266, Full "Clear data" never erases global CDC shared state GitHub, 20 July 2026. Checked 2026-08-29.
Airbyte issue 31312, No records being synced via CDC for large WAL GitHub, 2023. Checked 2026-08-29.
Debezium PR 7687, DDD-53: trigger-less outbox polling connector, proposal only GitHub, opened 18 July 2026, closed unmerged 14 August 2026. Checked 2026-08-29.
Airbnb, SpinalTap GitHub, master branch. Checked 2026-08-29.
Andreas Andreakis and Ioannis Papapanagiotou, DBLog: A Watermark Based Change-Data-Capture Framework arXiv:2010.12597, 23 October 2020. Checked 2026-08-29.
Martin Kleppmann, Alastair R. Beresford and Boerge Svingen, Online Event Processing: Achieving consistency where distributed transactions have failed Communications of the ACM 62(5), pages 43–49, May 2019. Checked 2026-08-29.
Sharma et al., Wormhole: Reliable Pub-Sub to Support Geo-replicated Internet Services NSDI '15; summarised by The Morning Paper, 14 May 2015. Checked 2026-08-29.
Gunnar Morling, Practical Change Data Streaming Use Cases With Apache Kafka and Debezium QCon San Francisco, 12 November 2019, 65 slides. Checked 2026-08-29.
Artem Livshits, Atomic Dual-write Recipes with Kafka Two Phase Commit (KIP-939) Current 2024, Confluent. Checked 2026-08-29.
Chris Egerton, Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors Kafka Summit London 2023. Checked 2026-08-29.
Wade Waldron, Understanding the Dual-Write Problem and Its Solutions Confluent blog, 29 May 2024. Checked 2026-08-29.
Confluent Developer, Designing Event-Driven Microservices: The Listen to Yourself Pattern Confluent Developer course, undated. Checked 2026-08-29.