Tiered Storage

A log-based streaming system historically bounded retention by broker local disk. Retaining months of history meant provisioning enormous storage on machines sized for throughput — paying for expensive compute-attached storage to hold data almost nobody reads — and it made broker replacement slow, since replacing a node meant copying all of its data.

Tiered storage separates the two. Recent segments stay on local disk for low-latency tail reads; older segments move to object storage and are fetched transparently when a consumer needs them.

Why it matters

The immediate effect is economic, and the important effect is architectural. When retaining a year of history is affordable, the log stops being a transient buffer and becomes a replayable system of record.

That changes what is possible: rebuilding a derived store, backfilling a new consumer from the beginning, and reprocessing after a bug all become routine operations rather than projects that depend on whether the data still exists. The log-as-source-of-truth pattern becomes practical rather than aspirational.

The operational effect is also substantial: rebalancing and broker replacement become fast, because most of the data does not move — which changes the cost of scaling and of hardware failure.

Implementation patterns

• Local retention sized for the tail-reading workload, which is almost all consumption, and remote retention sized for replay and compliance.
• Capacity planning split in two: throughput sizing for brokers, retention sizing for object storage. These are now independent variables, and treating them as one is the common sizing error.
• Expect tiered reads to be slow and to incur request pricing. A consumer replaying from the beginning has a completely different cost and performance profile from one reading the tail — and a replay that saturates object storage request quotas is a real and surprising failure.
• Throttle replay for the same reason as any backlog drain: the historical data will be consumed far faster than it was produced.
• Separate consumer groups for replay, so a backfill does not compete with live consumption.
• Monitor tiered read volume and cost, which is a line item that appears only when someone replays.
• Verify retention actually covers the reprocessing window you rely on, rather than assuming — this is the assumption that fails when a backfill is finally needed.

Industry example

Tiered storage became a headline capability of Kafka and its commercial distributions, alongside the KRaft work that removed the ZooKeeper dependency. Together they addressed the two largest operational objections to running a log at scale: two distributed systems to operate, and retention bounded by expensive local disk.

The same architecture appears wherever a log must serve both low-latency tail consumption and occasional historical replay — and the pattern generalises beyond streaming, being the same hot-cold tiering used in observability platforms, analytical storage and archival systems, for the same reason: the access distribution is enormously skewed toward recent data, and pricing storage uniformly for the whole range is wasteful.

Failure scenarios

• Local retention too small, so ordinary consumer lag pushes reads into the tiered layer and latency degrades for normal operation.
• Replay saturating object-storage request limits, producing a self-inflicted incident.
• Unthrottled replay competing with live consumption.
• Retention assumed rather than verified, discovered when a backfill needs data that expired.
• Cost surprise from an unmonitored replay of a large history.
• Treating long retention as a substitute for backups, which it is not — a log with a bad producer's data in it retains the bad data faithfully.
• Unbounded retention with no lifecycle policy, including for data with erasure obligations.

Trade-offs

Tiered storage makes the fast path slower for anything not in the local tier, and the boundary is a tuning decision that trades local storage cost against the latency of reads that fall through.

It also introduces object storage as a dependency of the streaming system, with its own availability characteristics and its own request-based pricing — which means a class of failure and a class of cost that did not previously exist.

And cheap long retention creates a governance obligation. Data retained for years is data subject to retention limits, erasure requests and access controls, and a log kept indefinitely because storage became cheap is a compliance exposure that nobody decided to accept.

The trade is tail-read latency and a new dependency in exchange for retention decoupled from broker cost — and for replay becoming a design tool. For a system that never replays and keeps a week of data, it adds complexity for nothing. For one that treats the log as a source of truth, it is what makes that treatment affordable.

Interview question

"We keep seven days of retention because longer was too expensive. Tell me what changes if we can affordably keep a year, what new failure modes we acquire, and what you would put in place before the first person replays from the beginning."