intermediate 3 min answer Multiple choice

A topic ingests 200 MB per second, replication factor 3, retained 30 days. Producers and consumers are spread across 3 availability zones with no rack awareness in the consumers. Roughly what does the storage footprint come to?

kafkacostretentionreplicationcross-az
Pick one
Show the full answer Hide the answer

The arithmetic, shown

Start from bytes per day, then multiply by the two factors people forget.

  • 200 MB/s × 86,400 s = 17.3 TB per day of unique data.
  • × 3 replicas = 51.8 TB per day on disk.
  • × 30 days = about 1.5 PB, before any compression ratio you negotiate with the format.

Why the other options fail

  • About 155 TB is 30 days at one replica. It is the number a capacity request usually contains, because replication is invisible in the throughput figure the team was given.
  • About 520 TB counts the replicas but only ten days, the shape of error you get when a retention setting in milliseconds is misread by a factor of three.
  • About 4.5 PB applies replication and the three-zone footprint as separate multipliers. The three copies are the three zones, and double-counting them is the most common over-estimate in cluster sizing reviews.

The assumption that dominates the error

Retention, by a wide margin. Storage is linear in days, and 30 days is a number that is almost always inherited rather than chosen. Ask what actually needs it: if the answer is "a consumer might need to replay", the real requirement is usually the worst realistic outage plus a margin, which is days rather than weeks. Cutting retention from 30 days to 7 removes about 1.2 PB. Compression is the second lever and it is bounded: 3× to 5× on JSON-like payloads with a columnar-friendly codec, far less on already-compressed binary.

The number nobody put in the estimate

Cross-zone traffic is usually the larger bill. With consumers unaware of rack placement, roughly two thirds of consumer fetches cross a zone boundary, and replication traffic between brokers crosses zones by design. At 200 MB/s ingest with three consumer groups, cross-zone egress is on the order of hundreds of MB/s continuously. At typical 2024-2025 cloud inter-AZ rates of about \(0.01 per GB in each direction, 100 MB/s sustained is roughly \)2,600 a month, and this line item routinely exceeds the storage it accompanies. Rack-aware fetching from the nearest replica, where the client and broker versions support it, is the change that removes most of this bill for least work.

What the number rules in or out

1.5 PB on broker-attached SSD is a large fleet whose size is set by storage rather than by throughput, which is the signal to move older segments to object storage and keep only recent segments local. The decision rule: if the disk footprint sizes your broker count, tiered storage pays; if throughput or connection count sizes it, tiering saves nothing.

It also rules out the reflex of raising replication factor to 4 or 5 for durability. That is another 500 TB and another stream of cross-zone replication traffic, to protect against simultaneous loss of three zones, which is not the failure the fleet actually experiences.

Common weak answers

  • "Storage is cheap." At this footprint it is the dominant line item, and it is recurring: the estimate is a monthly commitment, not a one-off purchase.
  • "Compression will handle it." Compression is already assumed in the vendor's throughput figure about half the time, so applying it again is double-counting. State whether your 200 MB/s is pre- or post-compression before anyone sizes anything.