Since 2023 a generation of production systems has replaced replicated local disks with object storage as the durability layer: Kafka-compatible streams, serverless Postgres, search engines, observability stores. This guide reconstructs the shape they converge on, the argument the Kafka community had about it for ten and a half months, what it costs, and where it has already broken in production.
Keeping bytes alive when machines die, priced two ways, and the pattern that moved the answer into someone else's building.
Strip the technology names and the problem is old: data must survive the death of the machine that wrote it. The classical answer is three copies on three disks in three failure domains, maintained by the database itself. In the cloud, that answer bills you twice. The disks cost roughly four times what the equivalent object storage does before you even triple them (EBS gp3 at $0.08/GiB-month against S3's $0.02 post-replication, per WarpStream's 2023 analysis), and the replication traffic between availability zones is metered per gigabyte. WarpStream's published TCO worksheet makes the second charge vivid: a 3-AZ open-source Kafka cluster at $20,252 a month, of which $14,765 is inter-zone networking, nearly five times the $2,961 their entire replacement runs on.
So a generation of systems stopped maintaining the three copies and handed the job to the provider's object store, which already keeps them. WarpStream shipped a Kafka-compatible service with no data-path disks in 2023; Confluent's Freight clusters followed with a direct-write mode in 2024; Aiven's KIP-1150 put the same idea into Apache Kafka itself, surviving a ten-and-a-half-month argument to a 9-binding-vote acceptance on March 2, 2026. Outside streaming the same move shows up in Neon's Postgres storage, turbopuffer's search engine, Datadog's Husky, and SlateDB, an embedded LSM whose only storage is a bucket. The sources use three names for one pattern: "Zero Disk Architecture", "diskless", "object-storage-native". This guide calls it delegating durability, because that is the actual transaction.
Read the designs side by side and the disks never disappear. Every system in the public record that "deleted its disks" kept a small, strongly consistent ordering core, and it is usually the most conventional component imaginable. Aiven's diskless Kafka is leaderless at the data layer, and its FAQ then says the quiet part plainly: "there is still a central coordinator, specifically the 'Batch Coordinator,' for metadata management, which is currently backed by PostgreSQL." To remove Kafka's disks, they added a relational database. What the pattern deletes is the fleet of data-path disks; what remains is coordination, and that remainder sets the latency floor, the availability ceiling, and more of the bill than the marketing suggests.
Scope. This guide covers systems that make object storage the primary durability layer for live, online writes: streams, OLTP, search, observability. It deliberately excludes lakehouse table formats and their query engines (Iceberg, Delta), tiered storage that only moves cold data, multi-region and multi-cloud designs, and self-hosted object stores. It also stays on the write and coordination paths; read-path cache design gets one section-worth of attention, not the full treatment it deserves.
Four components recur in every published implementation. Three are celebrated. The fourth is the one that pages you.
The stateless data plane. Because no node owns any data, any node can serve any partition. WarpStream's agents are explicitly "stateless and leaderless"; any agent can sequence for any topic or coordinate any consumer group. Datadog's Husky readers "are stateless and can be scaled up or down without issues". The operational dividend is real and measured: ShareChat reports running WarpStream with plain autoscaling and no StatefulSets, and puts the migration's savings at 58-60% against multi-AZ Kafka in their joint case study. Node replacement stops being a data operation; Inkless notes a returning broker is "immediately back in ISR" because there is nothing to catch up.
The batching buffer. Object stores price by request, so nobody writes records; everybody writes merged batches. Inkless flushes every 250 ms or 8 MB, whichever comes first, and combines produce requests from many clients, and many partitions, into one object. WarpStream's default batch timeout is likewise 250 ms. SlateDB's RFC 0008 states the reason at the level of principle: "Group commit is essential." This buffer is not an optimisation of the design. It is the design; every latency and cost property downstream is set by it, which is why section 03 treats the flush interval as a first-class architectural decision.
The ordering core. An object store will keep your bytes but it will not tell two writers who came first at any useful rate. Every system therefore keeps a strongly consistent component that assigns order, and the four published implementations span an entire design space. Inkless commits batch coordinates "in a linear order" to PostgreSQL. WarpStream runs a proprietary replicated state machine in its control plane, and its docs are precise about the consequence: after the S3 upload succeeds, "the file does not exist in WarpStream until this step is completed even though the data has already been made durable in object storage." Neon refuses to put even that much on the write path: a commit is a Paxos quorum of safekeepers on NVMe, and object storage only ever receives history asynchronously. And since AWS shipped conditional writes (If-None-Match, August 2024), the store itself can be the coordinator: SlateDB fences writers through compare-and-swap on a manifest, and turbopuffer "even drives consensus with object storage through conditional writes", per their CMU talk. The catch is throughput: turbopuffer measured a single contended object at roughly 5 compare-and-swap writes per second, since strong consistency forces each ~200 ms operation to be non-overlapping in time.
The async compactor. Batching for cheap writes means one object carries many partitions, which is a terrible layout for reading one partition. Every system runs a background process that re-merges objects into read-friendly ones: Inkless brokers merge recent objects "to improve the locality of data in each partition", Husky's compactors do streaming k-way merges over fragments, SlateDB compacts its L0 into sorted runs. Compaction is where the read path's economics are actually decided, and it is also a second full client of both the object store and the ordering core.
One more element is easy to miss because it lives in client configuration rather than in a service: zone-aware routing. Inkless hashes each partition to one broker per availability zone and steers clients to their local zone, because a diskless cluster that lets clients talk across zones has quietly reinstated the per-gigabyte bill it was built to remove. The cross-zone charge does not go away; it goes to whoever stops paying attention to it.
Where implementations genuinely diverge: Neon is the documented dissent. Its founding decision was to split the WAL service (consensus, disks, quorum) from the page service, and its Lakebase-era write-up keeps the boundary sharp: "queries do not read object storage", and a read from it "can take hundreds of milliseconds", so S3 appears only inside the pageserver when reconstructing history. Snowflake's NSDI '20 paper is the ancestor of the whole pattern, S3 for persistence with local ephemeral storage recovering the performance, and Confluent's Kora paper documents the half-step, tiering cold segments to object storage while the write path stays on replicated disks. The 2023-2026 wave is best read as those two ideas colliding on the write path itself.
Three forks recur in every design record, and the Kafka community spent most of 2025 arguing all three in public.
On replicated disks, latency is an engineering property; you buy better hardware once and tune. On object storage, producer latency is a metered rate: halving the flush interval doubles the PUT count, which WarpStream's own tuning docs describe plainly as "a tradeoff between cost and latency". Budget latency like a line item, and re-quote it whenever the provider's request pricing changes, as it did in April 2025.
The third decision has a public history worth reading in full, because it is rare to watch an open-source community adjudicate an architecture in writing. KIP-1150 was published in April 2025; within months Slack's KIP-1176 proposed the opposite shape (keep leaders, push the active segment to fast storage), AutoMQ's KIP-1183 proposed a third, and the discussion stalled badly enough that an August 2025 wiki document, "The Path Forward for Saving Cross-AZ Replication Costs KIPs", existed just to summarise the deadlock. Jack Vanlightly's October 2025 essay called it what it was: "the Kafka project finds itself at a fork in the road where choosing the right path forward for implementing S3 topics has implications for the long-term success of the project." The resolution was synthesis rather than victory: KIP-1150's second revision absorbed KIP-1176's idea by delegating long-term storage to the existing tiered-storage abstraction, KIP-1183's authors withdrew in its favour, and the vote passed in March 2026 with the implementation KIPs (1163, 1164) still being argued. Note what the accepted KIP also rejected, in writing: dropping classic topics entirely, because low-latency workloads still need disks, and the do-nothing option, because the cross-AZ bill is the dominant complaint of cloud Kafka operators.
| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Durability point | Ack after object-store PUT | Quorum of own disks first | Deletes stateful fleet + inter-AZ fees | p99 budget < ~100 ms; OLTP commit path | KIP-1150, Neon |
| Ordering | Dedicated coordinator | CAS on the store | CAS caps at ~5 writes/s per object | Single writer or rare ordering ops | turbopuffer, Inkless FAQ |
| Flush cadence | ~250 ms / 8 MB | Disk-like cadence | Cost grows exponentially below ~10 ms | Fast-tier pricing moves (Apr 2025) | Inkless FAQ, AWS |
| Storage tier | Standard S3 | S3 Express One Zone | Single-AZ durability; storage premium | Latency-sensitive WAL only; post-cut pricing | Vanlightly 2023, WarpStream 2024 |
| Long-term layout | Delegate to tiered-storage abstraction | New compaction subsystem in KIP-1150 v1 | Reuse beat reinvention in review | You are not inside Kafka; then you own compaction | Vanlightly |
| Client routing | Zone-aware, deterministic per-AZ owner | Any client, any broker | Otherwise cross-AZ fees return via clients | Single-AZ deployments | Inkless |
| Coordinator placement | 3 AZs, replicated, always | Coordinator in one zone | "You still need at least three AZs" for brokers and coordinator | Never, per every published design | Inkless FAQ |
Four failure classes cover every published incident this hunt found, and none of them is "the object store lost my bytes".
The striking absence first: no public postmortem in this corpus attributes an incident to data loss inside the delegated layer, an object acknowledged and later gone. Every published failure is an availability or coordination failure. That reads two ways, either the durability delegation genuinely works, which the providers' design points support, or such loss would be silent and nobody would have noticed, which Inkless concedes in its FAQ ("data would indeed be lost"). Both readings argue for the same control: an independent reconciliation job that compares the ordering core's index against what is actually in the bucket. Nobody's architecture page shows that job. Build it anyway.
What does break is instructive, because it maps exactly onto what the pattern kept and what it newly depends on. A sequence view of the common write path shows the seam every incident below runs through:
The fourth class appears in design docs as a fear rather than in postmortems as an event: the configuration change that multiplies request count without breaking anything. Aiven documents the cliff (sub-10 ms intervals, sub-100 KB objects, cost "can exceed the cost of Classic topics"); SlateDB's RFCs treat PUT cost as a failure mode to design against. No one has published the incident review for a surprise six-figure request bill, which almost certainly means it is being written up as a finance story, not an engineering one. Treat request-count-per-byte as an SLI with an alert, the way you treat error rate.
Sources: Inkless FAQ, SlateDB RFC 0008
Everything quantitative this hunt produced, with status: measured, claimed, or derived. Prices move; every row carries its date.
| Metric | Value | Status | Context | As of | Source |
|---|---|---|---|---|---|
| S3 Standard storage | $0.023/GB-mo | Vendor price | us-east-1, first 50 TB; already replicated | 2026-09 | AWS pricing |
| S3 Standard PUT | $0.005/1,000 | Vendor price | The unit your flush interval buys | 2026-09 | AWS pricing |
| EBS gp3 storage | $0.08/GiB-mo | Reported | Pre-replication; x3 for a replicated system | 2023 | WarpStream |
| Inter-AZ share of Kafka TCO | $14,765 of $20,252/mo | Claimed (worked example) | 3-AZ OSS Kafka; replacement infra $2,961/mo | 2023 | WarpStream TCO |
| Produce p99, standard S3 | ~400 ms | Reported | 4 MiB PUT p99; default 250 ms batching | 2024 | WarpStream docs |
| Produce, S3 Express One Zone | 33 ms med / 50 ms p99 | Vendor benchmark | Lowest-latency configuration | 2024 | WarpStream |
| End-to-end, S3 WAL vs EBS WAL | ~500 ms vs <10 ms | Vendor figure | Same system, pluggable WAL backends | 2025 | AutoMQ docs |
| S3EOZ price cuts | -31% / -55% / -85% | Vendor price | Storage / PUT / GET, effective 2025-04-10 | 2025-04 | AWS |
| CAS throughput on one object | ~5 writes/s | Measured | ~200 ms/op, non-overlapping by consistency | 2025 | turbopuffer |
| Warm vs cold query | p50 8 ms vs p90 444 ms | Reported | NVMe/RAM cache hit vs S3 read, 1M vectors | 2025 | turbopuffer |
| Migration savings | 58-60% / 21-27% | Case study | vs multi-AZ / single-AZ Kafka, logging workload | 2025 | ShareChat |
| Per-prefix request ceiling | 3,500 PUT / 5,500 GET per s | Vendor documented | After S3 partitions the prefix; gradual | 2026-09 | AWS docs |
| Cost of halving flush interval | ~2x PUT spend | Derived | PUT count scales inversely with interval | 2026-09 | arithmetic below |
A model small enough to keep in your head. One writer flushing every 250 ms makes 4 PUTs a second, about 10.5 million a month, which at $0.005 per thousand is roughly $52 a month (derived; 4 x 2,629,800 seconds x $0.000005). Drop the interval to 25 ms for snappier producers and the same writer costs ~$520 a month before it has stored a byte; run 40 such writers and the request bill alone clears $20k a year. The two variables that dominate everything are flush interval and writer count; storage volume, the number everyone models first, is usually third. What this model excludes: GET traffic (read-path and compaction), the ordering core's own infrastructure, and cross-AZ charges you reintroduce through careless client routing.
The 24x and 5x headline ratios and the TCO worksheet are WarpStream's own, published while selling the alternative; no independent replication of that worksheet exists in the public record. ShareChat's percentages are the strongest cost evidence here because a customer put their name on them. The latency figures are consistent across three independent implementations (WarpStream, AutoMQ, Aiven's defaults), which is why this guide treats "hundreds of milliseconds on standard S3" as corroborated rather than claimed.
Every source behind this page, graded. The full ledger with per-claim
quotes ships alongside as sources.md.
Credential rotation for the R2 Gateway deployed to the wrong environment; 100% of writes and ~35% of reads failed for 1h07m. Root cause: one omitted CLI flag.
An operator action against one phishing report disabled the whole R2 Gateway for 59 minutes; every R2-dependent product went with it.
A race between DNS automation instances emptied DynamoDB's endpoint records: ~3h of metadata-store unavailability, ~15h cascade. The exact failure shape a zero-disk system's ordering core is exposed to.
A mistyped capacity-removal command took out the index subsystem holding metadata for every object in the region; recovery required restarts not exercised for years.
The KIP-1150 reference implementation's own docs: PostgreSQL batch coordinator, 250 ms / 8 MB batching, three-AZ DR requirement, and the exponential cost cliff below 10 ms flush intervals.
The recorded argument: three simultaneous KIPs, a two-month stall, and the thread that forced a synthesis. The closest thing this domain has to a rejected-PR record.
The transactions RFC whose review thread surfaced the commit-vs-durability split on object storage and spawned RFC 0008.
The accepted proposal, with rejected alternatives recorded: dropping classic topics (low-latency workloads still need disks), per-cluster diskless, doing nothing.
The competing design: keep partition leaders and replication semantics, route the active segment through fast shared storage so followers stop crossing AZs. ~40% claimed savings.
Two records of the same physics: S3's floor latency defeats any flush interval, so RFC 0009 adds a second, faster object store for the WAL alone; RFC 0008 designs the buffering and read-only fallback for when the store misbehaves.
A named customer's measured outcome: 58-60% cheaper than multi-AZ Kafka, 21-27% cheaper than single-AZ, with autoscaled stateless agents replacing StatefulSets.
The cost case (24x storage ratio, the $20,252 worksheet) and the availability refinement: ack after durability, sequence lazily, so the metadata store leaves the critical path.
The documented dissent: commits are a Paxos quorum of NVMe safekeepers; object storage holds history and is never on the query path, because its reads take hundreds of milliseconds.
The measured ceiling of store-as-coordinator: ~5 conditional writes per second on one object at ~200 ms each, and the stateless-broker design that recovered 10x tail latency.
Writers upload to blob storage then commit to a metadata store that is "the strongly consistent source of truth"; readers and compactors are stateless around it.
The independent analyst's through-line: the 2023 disappointment with Express One Zone's pricing, the mechanics of Neon's quorum, and the 2025 essay framing the Kafka KIP contest as a fork with project-level stakes.
Within weeks of the If-None-Match launch: a working lock and leader-election scheme on bare S3, correctness argued from strong read-after-write consistency.
First-person account of the per-prefix quota's warm-up behaviour degrading a high-traffic workflow: limits start bucket-wide, isolation arrives only after S3 repartitions.
The ancestor: persistent data on S3, described as reliable and cheap but not fast, with local ephemeral storage recovering performance. The 2023-2026 wave applies the same bargain to the write path.
The half-step, documented rigorously: write to local disks, tier aging segments to object storage, shrink local volumes to ease rebalancing. Best industry paper.
Eskildsen on the full design: consensus through conditional writes, tiered NVMe/RAM cache for sub-10 ms warm queries, and the economics that make search viable at trillion-document scale.
The case study presented by the team that ran the migration, with the operational detail (agent roles, autoscaling behaviour) the written version compresses.
The three provider moves that reshaped this space: If-None-Match (Aug 2024), the April 2025 Express One Zone cuts, and the documented per-prefix request model.
Kora evolved to a direct-write mode, bypassing local storage and broker replication, sold as "up to 90% cheaper" for relaxed-latency workloads. A claim, not a measurement, and labelled as such here.
Seven rungs from an evening toy to something with the production pattern's actual failure modes. Any S3-compatible store works; real S3 teaches the real latency.
A key-value store that PUTs and GETs single JSON objects, timing every operation and printing a running histogram.
Done when: you have a day of p50/p99 numbers for 1 KB and 4 MB objects. Teaches: the latency floor no flush interval can beat, per SlateDB RFC 0009.
A write-ahead log that buffers records and flushes every N ms or M bytes. Emit a counter of PUTs and compute the monthly dollar cost at list price alongside your latency histogram.
Done when: one chart shows cost and p99 moving oppositely as you sweep N from 1000 ms to 10 ms. Teaches: latency as a line item, the pattern's central trade.
Two competing writer processes, one manifest object, If-None-Match and If-Match arbitrating which one holds the pen. Kill and restart them randomly.
Done when: an hour of kill-loops produces zero interleaved writes, and you can state the CAS ops/sec you achieved. Teaches: the store as coordinator, and its ~5 ops/s ceiling.
Multiple writers upload batches concurrently; a coordinator (SQLite or Postgres) assigns offsets to uploaded objects in commit order. Now kill a writer after its PUT succeeds but before its commit lands.
Done when: your reconciler finds the orphaned object and either admits or deletes it, deterministically. Teaches: durable-but-nonexistent, the seam in figure 5.
A reader that serves from a local disk cache, fetching misses from the store with readahead. Measure cold-start against warm steady state.
Done when: you can show the cold/warm gap (expect two orders of magnitude, per turbopuffer's 8 ms vs 444 ms). Teaches: why cache eviction policy is availability policy here.
A proxy in front of the store that injects 503 SlowDown bursts, 5-minute total outages, and latency spikes. Watch your buffer grow; implement the bounded-buffer to read-only degradation SlateDB's RFC 0008 specifies.
Done when: a 5-minute store outage loses zero acknowledged writes and recovers unattended. Teaches: that your real durability guarantee is buffer-depth times outage-length.
A background merger that rewrites small multi-stream objects into per-stream sorted ones, updating the coordinator transactionally, while writes continue. Then re-run rung 6's chaos against it.
Done when: read amplification drops measurably and chaos still loses nothing. Teaches: compaction as a second full tenant of both store and coordinator.
The queries that found this material, grouped by what they surface. The domain vocabulary ("diskless", "zero disk", "batch coordinator") is the key that opens it.
KIP-1150 diskless topics rejected alternatives"batch coordinator" PostgreSQL diskless kafkasite:github.com slatedb rfcs synchronous commit durabilitysafekeepers why not write WAL directly to S3"zero disk architecture" S3 inter-AZ costS3 "SlowDown" 503 rate limit prefix production "we"cloudflare R2 outage postmortem credential rotation"aws.amazon.com/message" outage summary DynamoDB DNSobject storage queue conditional writes "writes per second"warpstream TCO "inter-zone networking" breakdownS3 Express One Zone price reduction PUT requestkafka "cost calculator" cross-AZ replication brutalsharechat warpstream logging case study savingshusky datadog object storage metadata store compactionturbopuffer CMU talk object storage search