Data Platforms 03 Sep 2026 30 min read

The disks never disappear: what "diskless" systems actually delete

How production systems replaced replicated local disks with object storage as the durability layer, what the ordering core they all kept costs, and where the pattern has already broken.

Read the field guide → The evidence ledger

Reconstructs the zero-disk pattern across nine production systems (WarpStream, Kafka's KIP-1150, Confluent Freight, Neon, turbopuffer, Datadog Husky, SlateDB, AutoMQ, Snowflake's ancestor) from their own design records, the ten-month Kafka KIP contest, four postmortems, and published cost figures. A reader leaves able to argue where the durability point and ordering core belong for a given latency budget, and what the flush interval costs in dollars.

The finding that surprised me

Every 'diskless' system in the public record kept a small, strongly consistent, often disk-backed ordering core; Aiven's diskless Kafka removed the brokers' disks by adding a PostgreSQL database, and that remainder sets the latency floor, the availability ceiling, and much of the bill.

What you get out of it

The disks move and shrink rather than disappear: what gets deleted is the data-path fleet, what remains is coordination (PostgreSQL, a replicated state machine, a Paxos quorum, or a CAS'd manifest).
On object storage, producer latency is a metered rate, not an engineering property: halving the flush interval doubles the PUT bill, and Aiven documents the cost curve going exponential below ~10ms/100KB.
S3's August 2024 conditional writes turned the store itself into a viable coordinator, but turbopuffer measured the ceiling at ~5 sequenced writes per second per object, so the external coordinator is now a throughput decision rather than an existence requirement.
No public postmortem attributes an incident to data lost inside the delegated layer; every published failure is availability or coordination (Cloudflare R2 twice in 2025, DynamoDB DNS in October 2025, S3's index in 2017), which argues for an independent bucket-vs-index reconciliation job nobody's architecture page shows.
The realistic, customer-attested savings band is 20-60% (ShareChat), not the 10x of vendor headlines; the spread is mostly your AZ topology.

Scope

Why this, now. KIP-1150's acceptance in March 2026 made diskless topics official Apache Kafka direction, and the argument that got it there is the best-documented architecture debate in years.

What it does not cover. Lakehouse table formats and query engines, cold-only tiered storage, multi-region and multi-cloud designs, self-hosted object stores, and deep read-path cache design.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.

Keep digging

Other field guides

Data Platforms

The disks never disappear: what "diskless" systems actually delete

What you get out of it

Scope

Other field guides

The cutover is the easy part

A cache miss is a protocol, not an event

A timeout is not an answer: making retried requests safe