Evidence ledger
28 sources
Checked 29 Aug 2026
Evidence ledger
One row per claim in You cannot delete the dual write, only move it: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: You cannot delete the dual write, only move it Topic: how production systems keep a database write and the event that announces it in agreement. Research date: 2026-08-29. All links fetched on 2026-08-29.
Tiers follow references/evidence.md: postmortem, source, adr, casestudy, blog,
paper, talk, vendor.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | PagerDuty | August 28 Kafka Outages: What Happened and How We're Improving | postmortem | 2025-09 | 2026-08-29 | https://www.pagerduty.com/eng/august-28-kafka-outages-what-happened-and-how-were-improving/ | An outbox converts a broker outage into a backlog and a duplicate-delivery event, not a loss event. | "No previously accepted events or data were lost during or after the incident." / "affected customers may have received duplicate webhooks." |
| 2 | PagerDuty | August 28 Kafka Outages | postmortem | 2025-09 | 2026-08-29 | https://www.pagerduty.com/eng/august-28-kafka-outages-what-happened-and-how-were-improving/ | The trigger was a producer-lifecycle bug, not a data-consistency bug; the consistency machinery held while the transport collapsed. | "nearly 4.2 million extra producers per hour at peak. This is 84 times higher than our typical number of new producers" |
| 3 | PagerDuty | August 28 Kafka Outages | postmortem | 2025-09 | 2026-08-29 | https://www.pagerduty.com/eng/august-28-kafka-outages-what-happened-and-how-were-improving/ | Blast radius and duration of the two incidents. | First incident 03:53 UTC to 10:10 UTC; second 16:38 UTC to 20:24 UTC; "18.87% of create requests returned 502 errors at peak" |
| 4 | Cloudflare | Cloudflare incident on November 14, 2024, resulting in lost logs | postmortem | 2024-11-26 | 2026-08-29 | https://blog.cloudflare.com/cloudflare-incident-on-november-14-2024-resulting-in-lost-logs/ | In an at-least-once event pipeline, the buffer tier is the component that turns a configuration error into permanent loss. | "about 55% of the logs we normally send to customers were not sent and were lost"; Buftee went from "40 million buffers globally" to roughly 40 times that |
| 5 | Cloudflare | Cloudflare incident on November 14, 2024 | postmortem | 2024-11-26 | 2026-08-29 | https://blog.cloudflare.com/cloudflare-incident-on-november-14-2024-resulting-in-lost-logs/ | Scale of the pipeline and the untested failsafe. | "4.5 trillion individual event logs" pushed daily; logreceiver processes "about 45 PB (uncompressed) of customer event logs each day"; the team "neglected to regularly test that the broader system was capable of handling a fail open event" |
| 6 | GitHub | October 21 post-incident analysis | postmortem | 2018-10-30 | 2026-08-29 | https://github.blog/news-insights/company-news/oct21-post-incident-analysis/ | When two stores each hold writes the other does not, the recovery cost is reconciliation, and it is measured in hours not seconds. | "Because the database clusters in both data centers now contained writes that were not present in the other data center, we were unable to fail the primary back over"; "one of our busiest clusters had 954 writes in the affected window" |
| 7 | GitHub | October 21 post-incident analysis | postmortem | 2018-10-30 | 2026-08-29 | https://github.blog/news-insights/company-news/oct21-post-incident-analysis/ | Event backlogs outlive the incident that produced them. | "over five million hook events and 80 thousand Pages builds queued"; degradation lasted 24 hours 11 minutes |
| 8 | Andreas Andreakis (author of Netflix DBLog) | Change-Data-Capture Doesn't Solve Dual-Writes | blog | 2026-08-05 | 2026-08-29 | https://aandreakis.com/posts/change-data-capture-doesnt-solve-dual-writes | The central thesis of this guide: CDC relocates the dual write rather than removing it. | "The remedy for dual writes is itself implemented as a dual write." / "CDC moves this crash window downstream and puts replayable history behind it. It does not close the window for side effects." |
| 9 | Apache Kafka | KIP-618: Exactly-Once Support for Source Connectors | adr | 2023-06-07 (accepted) | 2026-08-29 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors | Kafka's own design record confirms the relay-level dual write: records are produced, then offsets are written, and the gap between them is the duplicate window. | "without framework-level adaptations, exactly-once delivery for source connectors remains impossible"; the framework "periodically writes task offsets to an internal Kafka topic after records reach Kafka" |
| 10 | Apache Kafka | KIP-939: Support Participation in 2PC | adr | 2024-07-23 (accepted) | 2026-08-29 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC | Two-phase commit is being deliberately reintroduced to Kafka for exactly this problem, with its costs written down. | "Dual write is easy to implement but means that whenever there is a failure there is a high likelihood that the log and the database will diverge." / "The transaction is never aborted automatically" / read_committed consumers "cannot consume past the ongoing transaction" |
| 11 | Apache Kafka (JIRA) | KAFKA-15370: Support Participation in 2PC (KIP-939) | source | created 2023-08-17, updated 2026-08-13 | 2026-08-29 | https://issues.apache.org/jira/browse/KAFKA-15370 | Three years after acceptance the feature has not shipped. | Status "Open", Resolution "Unresolved", Fix Version "4.5.0", updated 13/Aug/26 |
| 12 | Apache Kafka | Apache Kafka 4.0.0 Release Announcement | vendor | 2025-03-18 | 2026-08-29 | https://kafka.apache.org/blog/2025/03/18/apache-kafka-4.0.0-release-announcement/ | KIP-939 is absent from the 4.0 headline feature set. | Headline KIPs listed are 848, 932, 966, 996, 890, 1102, 653; KIP-939 is not mentioned |
| 13 | Apache Flink | FLIP-319: Integrate with Kafka's Support for Proper 2PC Participation | adr | 2023-08-18 | 2026-08-29 | https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710 | The most widely deployed exactly-once Kafka sink has a documented data-loss mode and works by bypassing the protocol. | "Data loss can occur when Kafka aborts a successfully checkpointed transaction due to timeout." / the implementation "relies heavily on Java reflection in order to bypass the Kafka transaction protocol" |
| 14 | Zalando | Contributing to Debezium: Fixing Logical Replication at Scale | casestudy | 2025-12-19 | 2026-08-29 | https://engineering.zalando.com/posts/2025/12/contributing-to-debezium.html | The replication slot and the connector's offset store are two records of the same position, and they drift. | "we've always treated the PostgreSQL replication slot as the authoritative source of truth for stream position"; Debezium had "hard-coded the pgjdbc keepalive flush feature to disabled by setting withAutomaticFlush(false)" |
| 15 | Zalando | Contributing to Debezium | casestudy | 2025-12-19 | 2026-08-29 | https://engineering.zalando.com/posts/2025/12/contributing-to-debezium.html | Scale and duration of the deployment, and the WAL-growth failure it fixed. | "replication slots wouldn't advance without table activity, causing WAL to pile up until disk space ran out"; "hundreds of thousands of events per second across our 100+ Kubernetes clusters"; ran Debezium 2.7.4 "for nearly two years, processing billions of events with zero detected data loss" |
| 16 | Shopify | Capturing Every Change From Shopify's Sharded Monolith | casestudy | 2021-03-12 | 2026-08-29 | https://shopify.engineering/capturing-every-change-shopify-sharded-monolith | Log-based CDC of business tables is viable at very large scale, and its cost is schema coupling. | "~150 Debezium connectors across 12 Kubernetes pods"; "400TB+ of CDC data in our Kafka cluster"; "~65,000 records/second on average, handling spikes up to 100,000 records/second"; "p99 latency is less than 10 seconds" |
| 17 | Shopify | Capturing Every Change From Shopify's Sharded Monolith | casestudy | 2021-03-12 | 2026-08-29 | https://shopify.engineering/capturing-every-change-shopify-sharded-monolith | Snapshotting is the operational tax of table-level CDC. | "Several tables in Shopify's Core monolith are too big to snapshot in any reasonable time-frame"; initial snapshots hold read locks for hours and block binlog ingestion |
| 18 | Trade Republic (Sadeq Dousti) | PostgreSQL + Outbox Pattern Revamped, Part 1 | blog | 2025-06-08 | 2026-08-29 | https://dev.to/msdousti/postgresql-outbox-pattern-revamped-part-1-3lai | A polled outbox degrades by five orders of magnitude through index bloat, and partitioning is the fix. | "it fetches close to 100 million rows from the heap, which takes over 18.5 seconds to execute. This means a slowdown 5 orders of magnitude"; after partitioning, "a consistent result of 1000 heap fetches and an execution time of 1-3 ms" |
| 19 | Zapier | Lessons from using the outbox pattern at scale | blog | 2026-03-30 (date from search listing; not shown on page) | 2026-08-29 | https://zapier.com/blog/lessons-from-using-outbox-pattern-at-scale/ | A local durable outbox on the write path buys availability and charges you storage operations. | "fifteen thousand events per second" at peak; 50 sharded SQLite files per pod; replaced by an S3 and SQS sidecar that "removes the local SQLite write from the hot emit path" |
| 20 | Gunnar Morling (Confluent, ex-Debezium lead) | Can Debezium Lose Events? | blog | 2023-11-14 | 2026-08-29 | https://www.morling.dev/blog/can-debezium-lose-events/ | Loss in log-based CDC is an operational condition, not a defect: it happens when retention expires before the reader catches up. | "In general, Debezium by itself should never miss any event."; loss occurs when "due to operational deficiencies portions of the database's transaction log get discarded before Debezium gets a chance to capture them"; MySQL binlog_expire_logs_seconds defaults to 2,592,000 seconds |
| 21 | Gunnar Morling | Can Debezium Lose Events? | blog | 2023-11-14 | 2026-08-29 | https://www.morling.dev/blog/can-debezium-lose-events/ | The Postgres safety mechanism trades one failure for the other. | Replication slots prevent loss by default but risk disk exhaustion; PostgreSQL 13's max_slot_wal_keep_size reintroduces loss risk if configured |
| 22 | SQUER (David Leitner) | Stop overusing the outbox pattern | blog | undated on page | 2026-08-29 | https://www.squer.io/blog/stop-overusing-the-outbox-pattern | A credible dissent: the outbox makes the database the bottleneck of an architecture chosen to avoid one. | The pattern "turns the database into the system's bottleneck, something we wanted to avoid with EDAs in the first place." |
| 23 | Clear Street | Designing Clear Street's First Transactional Outbox Pattern | blog | 2020-10-01 | 2026-08-29 | https://www.clearstreet.io/news/blog/transactional-outboxes | An outbox table read by CDC is the common production shape, and its guarantee is explicitly at-least-once. | 2PC rejected as "slow and, frankly, complicated to implement"; "you are guaranteed that a corresponding event with the details of what you changed are published to Kafka at-least once" |
| 24 | Airbyte (issue #82266) | Full "Clear data" never erases global CDC shared state | source | 2026-07-20 | 2026-08-29 | https://github.com/airbytehq/airbyte/issues/82266 | The offset-versus-slot divergence is not a Zalando idiosyncrasy; it is a reproducible failure with its own error string. | "Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future." |
| 25 | Airbyte (issue #31312) | [source-postgres] No records being synced via CDC for large WAL | source | 2023-10 | 2026-08-29 | https://github.com/airbytehq/airbyte/issues/31312 | A CDC reader can report success while capturing nothing, which is the worst failure shape available. | "WAL resume position 'null' discovered"; 900GB table, ~50GB WAL per day, ~25GB accumulated before sync |
| 26 | Debezium (PR #7687) | DDD-53: trigger-less outbox polling connector, proposal only | source | opened 2026-07-18, closed unmerged 2026-08-14 | 2026-08-29 | https://github.com/debezium/debezium/pull/7687 | The design space is still open: a proposal to detect changes without a replication slot was closed without technical objection, only relocated. | Maintainer response in full: "This DDD was moved to debezium/debezium-design-documents#51."; proposal cites "approximately 15 MB per million watched rows" |
| 27 | Airbnb | SpinalTap README | source | repo, master branch | 2026-08-29 | https://github.com/airbnb/SpinalTap/blob/master/README.md | An independent in-house CDC service converges on the same components as Debezium: source, destination, external state store. | "SpinalTap is a general-purpose reliable Change Data Capture (CDC) service"; state coordination via ZooKeeper, position tracked as initial_binlog_position with fileName, position, nextPosition |
| 28 | Netflix (Andreakis, Papapanagiotou) | DBLog: A Watermark Based Change-Data-Capture Framework | paper | 2020-10-23, arXiv:2010.12597 | 2026-08-29 | https://arxiv.org/abs/2010.12597 | Snapshot and stream can be interleaved without locks, which is the mechanism that makes backfill affordable. | "a watermark based approach that allows us to interleave transaction log events with rows that we directly select from tables"; "does not use locks and has minimum impact on the source"; "currently used in production by tens of microservices at Netflix" |
| 29 | Kleppmann, Beresford, Svingen | Online Event Processing: Achieving consistency where distributed transactions have failed | paper | CACM 62(5), May 2019 | 2026-08-29 | https://martin.kleppmann.com/2019/05/01/olep-cacm.html | The intellectual case for the log-first architecture, and the sentence the industry took as permission to stop using 2PC. | "Distributed transactions have failed as a mechanism for ensuring consistency across heterogeneous storage technologies in today's large-scale applications." |
| 30 | Facebook (Sharma et al., NSDI '15), via The Morning Paper | Wormhole: Reliable Pub-Sub to Support Geo-replicated Internet Services | paper | paper 2015; summary 2015-05-14 | 2026-08-29 | https://blog.acolyer.org/2015/05/14/wormhole-reliable-pub-sub-to-support-geo-replicated-internet-services/ | Reading the storage engine's own transaction log, with at-least-once delivery, is a decade-old production answer, not a new one. | "Wormhole publishers directly read the transaction logs maintained by the data storage systems"; "at least once delivery"; "over 35 GBytes/sec of updates at steady state across 5 trillion messages per day" |
| 31 | Gunnar Morling | Practical Change Data Streaming Use Cases With Apache Kafka and Debezium (QCon San Francisco 2019) | talk | 2019-11-12 | 2026-08-29 | https://speakerdeck.com/gunnarmorling/practical-change-data-streaming-use-cases-with-apache-kafka-and-debezium-qcon-san-francisco-2019 | The slogan that set the industry default, and the outbox design behind it. | Slide 10 and slide 60: "Friends Don't Let Friends Do Dual Writes"; outbox table design on slides 41 and 42 |
| 32 | Artem Livshits (Confluent) | Atomic Dual-write Recipes with Kafka Two Phase Commit (KIP-939), Current 2024 | talk | 2024 | 2026-08-29 | https://current.confluent.io/2024-sessions/atomic-dual-write-recipes-with-kafka-two-phase-commit-kip-939 | The 2PC revival is being presented to practitioners as the answer to exactly this problem. | "if failures happen Kafka and database will diverge"; the recipe guarantees "events are committed to Kafka iff changes are committed to the database, even in the presence of failures" |
| 33 | Chris Egerton (Aiven) | Exactly-Once, Again: Adding EOS Support for Kafka Connect Source Connectors, Kafka Summit London 2023 | talk | 2023 | 2026-08-29 | https://www.confluent.io/events/kafka-summit-london-2023/exactly-once-again-adding-eos-support-for-kafka-connect-source-connectors/ | Kafka Connect source connectors were not exactly-once until 3.3, which dates the gap precisely. | "Kafka Connect lacked this support for source connectors until version 3.3" |
| 34 | Confluent (Wade Waldron) | Understanding the Dual-Write Problem and Its Solutions | vendor | 2024-05-29 | 2026-08-29 | https://www.confluent.io/blog/dual-write-problem/ | The canonical vendor statement of the problem and the four-option menu everyone works from. | "The dual-write problem occurs when two external systems must be updated in an atomic fashion."; the outbox "works well for solving the dual-write problem but is restricted to transactional databases" |
| 35 | Confluent Developer | Designing Event-Driven Microservices: The Listen to Yourself Pattern | vendor | course page, undated | 2026-08-29 | https://developer.confluent.io/courses/microservices/the-listen-to-yourself-pattern/ | Listen-to-yourself removes the dual write by giving up read-your-writes, and the trade is stated plainly. | callers may query immediately and "it won't find what it is looking for"; validation failures after the caller disconnects create "additional inconsistencies in our system" |
Tier mix
postmortem 7 rows across 3 incidents (PagerDuty, Cloudflare, GitHub) · source 5 · adr 3 · casestudy 4 · blog 8 · paper 3 · talk 3 · vendor 2.
Distinct organisations represented: PagerDuty, Cloudflare, GitHub, Netflix, Zalando, Shopify, Trade Republic, Zapier, Clear Street, Airbnb, Airbyte, Debezium, Apache Kafka, Apache Flink, Facebook, Confluent, SQUER, Cambridge/independent authors. Eighteen.
Where the record runs out
- No published account quantifies duplicate rates. Every system here promises at-least-once and none publishes what the duplicate rate actually is under normal operation or after a rebalance. An architect sizing a deduplication store has no public number to work from.
- No public postmortem attributes a customer-visible incident to an outbox row that was committed and never published. The failure everyone designs against is the one nobody has written up. The published incidents are all about the transport, the buffer, the position record or the database underneath.
- KIP-939 has no production account at all. It was accepted in July 2024 and has not shipped. Everything written about it, including the Current 2024 session, describes intent.
- Trade Republic's part 2 and Clear Street's promised follow-up were not published. Both posts end by saying the harder material is coming. It did not come.