Evidence ledger
24 sources
Checked 03 Sep 2026
Evidence ledger
One row per claim in The disks never disappear: what "diskless" systems actually delete: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Checked 2026-09-03. Environment note: this session's egress allowed direct fetches of github.com / raw.githubusercontent.com only; all other pages were retrieved and quote-checked through the search tool's server-side fetch of the live page. Quotes marked (q) were copied from retrieved content; figures without (q) are stated in the retrieved content but were not copied verbatim.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | WarpStream | Cloud Disks are (Really!) Expensive | blog | 2023 | 2026-09-03 | https://www.warpstream.com/blog/cloud-disks-are-expensive | S3 storage is ~24x cheaper than the EBS it replaces once replication is counted | (q) "S3 costs \(0.02/GiB per month of storage, post-replication" vs "EBS GP3 volumes cost \)0.08/GiB per month"; "S3 is 24x cheaper than EBS" |
| 2 | WarpStream | How WarpStream Reduces Kafka Infrastructure Costs: A TCO Breakdown | blog | 2023 | 2026-09-03 | https://www.warpstream.com/blog/warpstream-benchmarks-and-tco | In their published example, inter-zone networking alone for 3-AZ OSS Kafka cost ~5x WarpStream's entire infrastructure | (q) "an equivalent 3 AZ Open Source Kafka cluster would cost over \(20,252/month"; inter-zone fees "\)14,765 vs. $2,961" |
| 3 | WarpStream | WarpStream architecture (docs) | blog | 2024 | 2026-09-03 | https://docs.warpstream.com/warpstream/overview/architecture | Agents are stateless; a replicated-state-machine metadata store sequences every file before it "exists" | (q) "the file does not exist in WarpStream until this step is completed even though the data has already been made durable in object storage" |
| 4 | WarpStream | Low Latency Clusters (docs) | blog | 2024 | 2026-09-03 | https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters | Batch timeout is an explicit cost/latency dial; default p99 produce ~400ms | (q) "double the value of the batch timeout from 250ms (the default) to 500ms, which is a tradeoff between cost and latency"; p99 of a 4 MiB S3 PUT ~400ms |
| 5 | WarpStream | The Art of Being Lazy(log): Delayed Sequencing | blog | 2025 | 2026-09-03 | https://www.warpstream.com/blog/the-art-of-being-lazy-log-lower-latency-and-higher-availability-with-delayed-sequencing | Ordering can be deferred after durability to cut latency and ride out metadata-store unavailability | Post describes acking after S3 durability, sequencing later ("delayed sequencing") |
| 6 | WarpStream + AWS | S3 Express One Zone Benchmark and TCO | blog | 2024 | 2026-09-03 | https://www.warpstream.com/blog/warpstream-s3-express-one-zone-benchmark-and-total-cost-of-ownership | With S3EOZ in lowest-latency config: median produce 33ms, p99 50ms | (q) "median produce latency of 33ms and p99 of 50ms"; "S3 Express PUTs are approximately ⅕th the cost of a regular S3 PUT" |
| 7 | WarpStream / ShareChat | Cost-Effective Logging at Scale: ShareChat's Journey to WarpStream | casestudy | 2025 | 2026-09-03 | https://www.warpstream.com/blog/cost-effective-logging-at-scale-sharechats-journey-to-warpstream | Measured migration outcome: 58-60% cheaper than multi-AZ Kafka, 21-27% vs single-AZ | (q) "WarpStream saved ShareChat 58–60% compared to multi-AZ Kafka and 21–27% compared to single-AZ Kafka" |
| 8 | ShareChat / Confluent Current | Cost-Effective Logging at Scale (talk) | talk | 2025 | 2026-09-03 | https://www.youtube.com/watch?v=4FCuYXTgLrI | Same migration presented by the practitioner team at Current 2025 | Conference recording of [7]; zero-ops autoscaling of stateless agents |
| 9 | Apache Kafka | KIP-1150: Diskless Topics | adr | 2025-04-16 | 2026-09-03 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150:+Diskless+Topics | The proposal, its rejected alternatives ("drop non-diskless topics", "per-cluster diskless", "do nothing") and rationale | (q) rejected because classic topics remain "appropriate for low latency use-cases" and removal "would be a backwards-incompatible change" |
| 10 | Apache Kafka | KIP-1176: Tiered Storage for Active Log Segment | adr | 2025 | 2026-09-03 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-1176:+Tiered+Storage+for+Active+Log+Segment | The competing design: keep leaders, push the active segment to fast storage; ~40% claimed savings | Slack's proposal; followers pull from the store instead of across AZs |
| 11 | Apache Kafka | The Path Forward for Saving Cross-AZ Replication Costs KIPs | adr | 2025-08 | 2026-09-03 | https://cwiki.apache.org/confluence/display/KAFKA/The+Path+Forward+for+Saving+Cross-AZ+Replication+Costs+KIPs | Three KIPs addressed the same problem simultaneously and discussion stalled for over two months | Wiki doc created to summarize KIP-1150/1176/1183 status after stall |
| 12 | Apache Kafka dev list | [DISCUSS] The Path Forward for Saving Cross-AZ Costs KIPs | source | 2025-08-05 | 2026-09-03 | https://www.mail-archive.com/dev@kafka.apache.org/msg151181.html | The recorded community argument over which architecture to adopt | Thread initiated 2025-08-05 on dev@kafka.apache.org |
| 13 | Aiven | KIP-1150 Accepted, and the Road Ahead | blog | 2026-03 | 2026-09-03 | https://aiven.io/blog/kip-1150-accepted-and-the-road-ahead | Vote passed 2026-03-02 with 9 binding votes, 10.5 months after publication | (q) "On March 2, 2026, the vote to accept Diskless topics into Apache Kafka passed with... 9 binding votes and 5 non-binding ones" |
| 14 | Aiven | The Hitchhiker's Guide to Diskless Kafka | blog | 2025 | 2026-09-03 | https://aiven.io/blog/guide-diskless-apache-kafka-kip-1150 | Mechanics of diskless topics: leaderless writes, batch coordinator as sequencer | BatchCoordinator "acts as a sequencer and persists the offsets in its backing store" |
| 15 | Aiven | Diskless 2.0: Unified, Zero-Copy Apache Kafka | blog | 2025 | 2026-09-03 | https://aiven.io/blog/diskless-unified-zero-copy-apache-kafka | KIP-1150 v2 merged ideas from the competing KIP-1176 rather than fighting it | Revision "paired Diskless with Tiered Storage" |
| 16 | Aiven (Inkless) | Inkless FAQ | source | 2025-2026 | 2026-09-03 (raw fetch) | https://github.com/aiven/inkless/blob/main/docs/inkless/FAQ.md | Removing Kafka's disks added a PostgreSQL coordinator; DR still needs 3 AZs; cost blows up at low latency | (q) "there is still a central coordinator... currently backed by PostgreSQL"; (q) "at extreme values (low latency <10ms and small objects <100kb)... Cost grows exponentially, and can exceed the cost of Classic topics"; (q) "you still need at least three Availability Zones (AZs) for the Kafka brokers and Batch Coordinator" |
| 17 | Aiven (Inkless) | Inkless Architecture | source | 2025-2026 | 2026-09-03 (raw fetch) | https://github.com/aiven/inkless/blob/main/docs/inkless/ARCHITECTURE.md | Reference write/read path: batch 250ms/8MB, commit batch coordinates to SQL, AZ-local deterministic caching | (q) "batch metadata... and location of the data (object ID & extent) are sent to a SQL database to be committed in a linear order" |
| 18 | SlateDB | RFC 0008: Synchronous Commit & Durability | adr | 2025 | 2026-09-03 (raw fetch) | https://github.com/slatedb/slatedb/blob/main/rfcs/0008-synchronous-commit.md | Object-store WAL fails more than disk; buffer in memory then fall to read-only | (q) "Writing WAL to S3 has a higher risk of permanent failures due to network instability compared to local disk"; max_unflushed_bytes breach "mark[s] the db into an read-only state" |
| 19 | SlateDB | RFC 0009: Separate Object Store for WAL | adr | 2025 | 2026-09-03 (raw fetch) | https://github.com/slatedb/slatedb/blob/main/rfcs/0009-separate-wal.md | No flush interval overcomes the store's floor latency; the fix is a second, faster store for the WAL only | (q) "Even if flush_interval is set to an extremely low value, it can't overcome the inherent latency of the underlying object store" |
| 20 | SlateDB | Introducing SlateDB / design docs | blog | 2024 | 2026-09-03 | https://slatedb.io/blog/introducing-slatedb/ | The embedded LSM-on-object-storage design: batching vs PUT cost, caching vs GET cost, manifest fencing | Design docs: "SlateDB batches writes" to mitigate "high write API costs (PUTs)"; single writer via "formally verified manifest fencing protocol" |
| 21 | SlateDB | PR #260 discussion (transactions RFC) | source | 2025 | 2026-09-03 | https://github.com/slatedb/slatedb/pull/260 | The durability-semantics argument that spawned RFC 0008 happened in a PR review | Merged 2025-07-10; durability split into follow-up RFC |
| 22 | Neon | WAL + S3: Lakebase storage for the era of agents | blog | 2025 | 2026-09-03 | https://neon.com/blog/wal-s3-lakebase-storage-for-the-era-of-agents | Neon refuses object storage on the commit and query paths; quorum of safekeepers first | (q) "a read from it can take hundreds of milliseconds"; queries "do not read object storage"; commit = quorum of safekeepers acks WAL |
| 23 | Neon | Architecture decisions in Neon | blog | 2022 | 2026-09-03 | https://neon.com/blog/architecture-decisions-in-neon | The separation of WAL service (consensus) from page service was a founding decision | WAL service = safekeepers running a consensus algorithm; pageserver uploads layers to S3 asynchronously |
| 24 | Jack Vanlightly | Neon: Serverless PostgreSQL (ASDS ch. 3) | blog | 2023-11-15 | 2026-09-03 | https://jack-vanlightly.com/analyses/2023/11/15/neon-serverless-postgresql-asds-chapter-3 | Independent analysis confirming Neon's Paxos-based safekeeper quorum in front of S3 | Commit defined by quorum of safekeepers, Paxos-based protocol |
| 25 | Jack Vanlightly | A Fork in the Road: Deciding Kafka's Diskless Future | blog | 2025-10-22 | 2026-09-03 | https://jack-vanlightly.com/blog/2025/10/22/a-fork-in-the-road-deciding-kafkas-diskless-future | The KIP contest was a real architectural fork with long-term project stakes; v2 delegated compaction to tiered storage | (q) "the Kafka project finds itself at a fork in the road where choosing the right path forward for implementing S3 topics has implications for the long-term success of the project" |
| 26 | Jack Vanlightly | S3 Express One Zone, not quite what I hoped for | blog | 2023-11-29 | 2026-09-03 | https://jack-vanlightly.com/blog/2023/11/29/s3-express-one-zone-not-quite-what-i-hoped-for | At launch, S3EOZ pricing and single-AZ durability disappointed the streaming use case | Title and analysis; context for the 2025 price cut in [34] |
| 27 | turbopuffer | turbopuffer: fast search on object storage | blog | 2023-2025 | 2026-09-03 | https://turbopuffer.com/blog/turbopuffer | Cold/warm split: warm p50 8ms from NVMe/RAM cache, cold ~p90 444ms (1M vectors) from S3 | (q) warm queries "p50=8ms"; cold queries "1M vectors p90=444ms"; object storage ~\(0.02/GB vs SSD cache ~\)0.10/GB |
| 28 | turbopuffer | How to build a distributed queue in a single JSON file on object storage | blog | 2025 | 2026-09-03 | https://turbopuffer.com/blog/object-storage-queue | Bare S3 compare-and-swap sequences ~5 writes/s per object (~200ms/op); a stateless broker restored 10x tail latency | (q) "only fit approximately 5 writes per second" because "strong consistency forces each write to be non-overlapping in time" at "~200ms per operation" |
| 29 | turbopuffer / CMU | Object Storage-native Database for Search (Simon Eskildsen, CMU DB seminar) | talk | 2026-03-09 | 2026-09-03 | https://www.youtube.com/watch?v=pqoRNwNaxfs | Consensus can be driven through object storage conditional writes; tiered NVMe/RAM cache gives sub-10ms warm queries (claims at talk scale: 1T+ docs) | Companion post: https://turbopuffer.com/blog/video-andy-pavlo-cmu ; "even drives consensus with object storage through conditional writes" |
| 30 | Datadog | Introducing Husky / Husky deep dives | blog | 2022-2023 | 2026-09-03 | https://www.datadoghq.com/blog/engineering/introducing-husky/ | Same shape at observability scale: stateless writers/readers/compactors around object storage plus a strongly consistent metadata store | (q) writers "upload events to blob storage... then commit the presence of these new files to the metadata store"; metadata store is "the strongly consistent source of truth" (https://www.datadoghq.com/blog/engineering/husky-deep-dive/, https://www.datadoghq.com/blog/engineering/husky-storage-compaction/) |
| 31 | Confluent | Kora: A Cloud-Native Event Streaming Platform For Kafka (VLDB '23, best industry paper) | paper | 2023 | 2026-09-03 | https://vldb.org/pvldb/vol16/p3822-povzner.pdf | The half-step: tiered storage (local disk for hot, object for cold) improves elasticity and cost without touching the write path | Tiering: data written to local disks first, moved to object store as it ages; smaller local volumes ease rebalancing |
| 32 | Snowflake / Cornell | Building an Elastic Query Engine on Disaggregated Storage (NSDI '20) | paper | 2020 | 2026-09-03 | https://www.usenix.org/conference/nsdi20/presentation/vuppalapati | The pattern's ancestor: persistent data in S3, performance recovered via local ephemeral storage/caching | "leverages reliable and cheap (but not fast) persistent remote storage (S3)" with "fast local ephemeral storage" for cache and intermediates; talk: https://www.youtube.com/watch?v=4CYNxisz3d0 |
| 33 | AWS | Amazon S3 adds new functionality for conditional writes | vendor | 2024-08 | 2026-09-03 | https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes | S3 gained If-None-Match PUT (and If-Match in 2024), enabling CAS directly on the store | PutObject "supports an optional If-None-Match header"; 412 on conflict |
| 34 | AWS | Up to 85% price reductions for Amazon S3 Express One Zone | vendor | 2025-04 | 2026-09-03 | https://aws.amazon.com/blogs/aws/up-to-85-price-reductions-for-amazon-s3-express-one-zone/ | The low-latency tier got 31% cheaper storage, 55% cheaper PUTs, 85% cheaper GETs (effective 2025-04-10) | PUT per 1,000 (≤512KB): \(0.0025 → \)0.00113 |
| 35 | AWS | Amazon S3 pricing | vendor | current | 2026-09-03 | https://aws.amazon.com/s3/pricing/ | Standard: \(0.023/GB-mo (first 50TB), \)0.005 per 1,000 PUT | Pricing page figures, us-east-1 |
| 36 | AWS | Best practices design patterns: optimizing Amazon S3 performance | vendor | current | 2026-09-03 | https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html | The invisible quota: 3,500 PUT / 5,500 GET per second per partitioned prefix, scaling is gradual with 503s meanwhile | "at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix" |
| 37 | Olsen Budanur | S3 is Lying to You: The Hidden Rate Limits That Degraded a High-Traffic Workflow | blog | 2024-2025 | 2026-09-03 | https://medium.com/@olsenbudanur/s3-is-lying-to-you-the-hidden-rate-limits-that-degraded-a-high-traffic-workflow-2b373b6f1119 | First-person account: limits start bucket-wide, per-prefix isolation arrives only after S3 repartitions under sustained load | Rate limits "start out as bucket-wide, not per-prefix"; low-traffic prefixes throttled by neighbours |
| 38 | AWS | Summary of the Amazon S3 Service Disruption (2017-02-28) | postmortem | 2017-03 | 2026-09-03 | https://aws.amazon.com/message/41926 | A typo'd capacity removal took out S3's index subsystem; full restart of subsystems not restarted "for many years" | (q) command "entered incorrectly and a larger set of servers was removed than intended"; index subsystem "manages the metadata and location information of all S3 objects in the region" |
| 39 | AWS | Summary of the Amazon DynamoDB Service Disruption in US-EAST-1 (2025-10) | postmortem | 2025-10 | 2026-09-03 | https://aws.amazon.com/message/101925/ | A DNS-automation race emptied DynamoDB's endpoint records; ~3h DynamoDB outage, ~15h cascade | Two DNS Enactors raced; stale plan applied then deleted by cleanup, wiping endpoint DNS records |
| 40 | Cloudflare | Cloudflare incident on March 21, 2025 | postmortem | 2025-03 | 2026-09-03 | https://blog.cloudflare.com/cloudflare-incident-march-21-2025/ | Credential rotation deployed to the wrong environment: 100% of R2 writes and ~35% of reads failed for 1h07m | (q) root cause "the omission of a single command-line flag, --env production" |
| 41 | InfoQ (reporting Cloudflare's incident report) | Cloudflare R2 incident, February 6, 2025 | postmortem | 2025-03 | 2026-09-03 | https://www.infoq.com/news/2025/03/cloudflare-incident-r2 | An abuse-remediation action disabled the whole R2 Gateway for 59 minutes; every R2-dependent product failed | Operator action "inadvertently disabled the R2 Gateway service instead of the specific endpoint"; Stream/Images 100% failure |
| 42 | Gunnar Morling | Leader Election With S3 Conditional Writes | blog | 2024-08 | 2026-09-03 | https://www.morling.dev/blog/leader-election-with-s3-conditional-writes/ | Conditional writes make S3 itself a viable lock/coordination service | Lock scheme: competing If-None-Match PUTs; strong read-after-write consistency guarantees visibility |
| 43 | AutoMQ | WAL Storage (docs) + KIP-1150 analysis | blog | 2025 | 2026-09-03 | https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/wal-storage | Pluggable WAL: S3 WAL ~500ms e2e cheap and elastic; EBS/regional WAL sub-10ms but zone-constrained | "S3 WAL (~500ms)" vs "EBS WAL (sub-10ms)"; EBS middle tier "requires all AutoMQ nodes to be in the same Availability Zone" (analysis: https://www.automq.com/blog/kafka-kip-1150-diskless-topics-better-solution) |
| 44 | Confluent | Freight Clusters are Generally Available | vendor | 2025 | 2026-09-03 | https://www.confluent.io/blog/freight-clusters-are-generally-available/ | Vendor claim: direct-write-to-S3 cluster type "up to 90% cheaper" for relaxed-latency workloads | Kora evolved to "direct write" mode, "bypassing local storage, and avoiding replication on the Kora brokers" |
| 45 | Stanislav Kozlovski | How KIP-1150 Diskless Topics makes Kafka stateless | blog | 2025 | 2026-09-03 | https://blog.2minutestreaming.com/p/diskless-kafka-topics-kip-1150 | Independent deep-dive; latency gap between disk writes and object-store round-trips called the hardest unsolved problem | KIP-1150 "still working through how to handle the latency gap" |
| 46 | Instaclustr | Kafka "Diskless": Proposals, Status & Insights | vendor | 2026 | 2026-09-03 | https://www.instaclustr.com/support/documentation/announcements/apache-kafka/kafka-diskless-proposals-status-insights/ | KIP-1150 is a "meta KIP"; implementation KIPs (1163 core, 1164 coordinator) still under debate as of March 2026 | KIP-1150 approved but "does not, by itself, deliver a production-ready implementation" |
Tier counts
postmortem 4 (rows 38-41) · source 4 (12, 16, 17, 21) · adr 5 (9, 10, 11, 18, 19) · casestudy 1 (7) · blog 18 · paper 2 (31, 32) · talk 2 (8, 29) · vendor 6 (33-36, 44, 46)
Absences worth recording
- No closed-unmerged pull request was found that carries the core argument; in this domain the recorded rejections live in KIP documents and the dev-list "Path Forward" thread instead (rows 9-12), plus KIP-1183's withdrawal in favour of KIP-1150.
- No public postmortem attributes a production incident to data loss inside a zero-disk system itself (an object acknowledged and then lost). Every published incident is an availability or coordination failure. Both readings, "the durability delegation works" and "such loss would be hard to detect", argue for independent reconciliation.