A telemetry log ingests 2 GB per second. The team proposes replacing broker-local disk with a design where brokers hold no data and every batch is written straight to object storage - the shape Pinterest describes for MemQ. Which cost lands first in production?

The deciding property

In a broker-disk design a produce request is acknowledged once the record is in the page cache of enough replicas, which is a local write plus one network hop. In an object-store design the durable act is a completed object write, and an object write to a regional object store is tens of milliseconds at best. Nothing in the design removes that; it only decides who waits for it.

So the system batches. Records accumulate in the broker's memory until the batch is large enough to be worth a PUT, and the acknowledgement latency becomes the batch fill time plus the PUT time. At 2 GB/s a 10 MB batch per partition fills quickly and the fill time is negligible; at 2 MB/s on a quiet topic the same batch size means seconds of delay. The latency a producer sees therefore depends on the throughput of its own partition, which is a property most teams have never had to reason about.

Pinterest's MemQ is the public example of this shape: open-sourced by Pinterest, described as running near-real-time workloads alongside its Kafka fleet since 2020, built on pluggable object storage under a decoupled packet-based write design, and reported at over 90% better cost efficiency than Pinterest's Kafka footprint in some cases.

What the design buys

Storage and serving stop being the same resource. Brokers hold no data, so a broker failure is not a replication event and adding capacity is not a rebalance. Retention becomes an object-lifecycle policy rather than a disk-sizing exercise, and the three-way replication that broker disks pay for is already inside the object store's price.

Why the other options fail

• The storage bill rising. Backwards. Object storage is cheaper per gigabyte than provisioned block storage, and the design also removes the explicit replication factor of three, so the storage line usually falls sharply. This option is the instinct that anything cloud-native costs more.
• Rewriting consumers because objects carry no offsets. Designs of this kind keep a small metadata or notification log that records each object as it lands, and consumers track their position in that log. The offset abstraction survives; it just points at objects instead of file segments.
• Doubling the partition count. Partition count was a proxy for disk and consumer parallelism. With storage external, throughput per partition is bounded by the object write path rather than by a disk, so if anything the pressure on partition count eases.

When this is the wrong answer

For a topic whose consumers need end-to-end latency under about 100 ms - a fraud decision, a dispatch loop, a trading signal - the batching floor is a hard disqualifier, and broker-local storage with tiered offload for old segments keeps the cost benefit without the latency. The split is per topic, not per cluster: the telemetry firehose and the dispatch events do not need the same design.