Partition Key Skew

Streaming systems scale by partition, and ordering is guaranteed only within a partition. Both properties depend on the key distribution, and real-world key distributions are rarely uniform.

The symptom is distinctive and confusing: total throughput plateaus well below capacity, most consumers are idle, one consumer's lag grows without bound, and adding consumers changes nothing — because a partition is consumed by exactly one member of a consumer group, so parallelism is capped by partition count and skew makes the effective cap lower still.

Causes are usually a large tenant in a multi-tenant system, a default or null value that a large share of records share, or a key with genuinely low cardinality relative to the partition count.

The fixes each cost something. Composite keys — appending a bounded random suffix to the hot key — restore distribution and destroy per-key ordering, which is acceptable only if the downstream computation is order-insensitive or re-aggregates. Splitting the hot tenant onto a dedicated topic or cluster preserves ordering and adds routing complexity. Increasing partition count helps only if the skew is from cardinality rather than from a genuinely dominant key, and it is disruptive because it changes the hash mapping for every existing key.

The design-time move is to model the key distribution before choosing it, using real data.