Kafka replaced ZooKeeper with KRaft and added tiered storage. What problems did each address, and how do they change how a platform is operated and sized?

KRaft: removing the external coordination dependency

The problem: cluster metadata — topics, partitions, configurations, the controller election — lived in ZooKeeper, a separate distributed system with its own operational model, its own failure modes and its own expertise requirement. Operating Kafka meant operating two distributed systems, and the metadata path had a scaling ceiling that limited partition counts.

What KRaft changes:

• One system to operate, with metadata held in an internal Raft-replicated log.
• Faster controller failover and faster metadata propagation, because the metadata is a log that brokers consume rather than a store they query.
• Substantially higher partition counts, since the metadata path is no longer the constraint — which directly raises the parallelism ceiling for large deployments.
• A simpler operational and security surface: one set of credentials, one upgrade path, one thing to monitor.

Operational implication: the migration itself is the significant work, and the ongoing benefit is a meaningful reduction in what a platform team must understand and maintain.

Tiered storage: decoupling retention from broker disks

The problem: retention was bounded by broker local disk. Retaining a log for a long period meant provisioning enormous local storage on brokers sized for throughput — paying for expensive storage attached to compute in order to keep data almost nobody reads. It also made rebalancing and broker replacement slow, since replacing a broker meant copying all its data.

What tiered storage changes:

• Recent data on local disk, older segments in object storage, with the broker fetching transparently.
• Retention decoupled from broker sizing, so months or years of history is affordable — which changes what the log is for: it becomes a replayable system of record rather than a transient buffer.
• Faster rebalancing and broker replacement, since most data does not need to move.
• Cheaper storage per byte by a large factor.

Operational implications: reads of tiered data are much slower and hit object-storage request pricing, so a consumer replaying from the beginning has a very different performance and cost profile from one reading the tail. Capacity planning splits into throughput sizing for brokers and retention sizing for object storage, which are now independent.

What this means architecturally

Long retention makes replay a design tool rather than an emergency measure. Rebuilding a derived store, backfilling a new consumer from the beginning, or reprocessing after a bug all become routine — which is what makes the log-as-source-of-truth pattern practical rather than aspirational.

And it lowers the bar for adopting a log at all, since the operational burden was historically a large part of the argument against it. The argument that remains is the workload one: if you do not need retention independent of consumption, multiple independent consumer groups, replay, or very high throughput, a table in the database you already have is still the better answer — and these improvements do not change that.