Key Skew

Partitioning by key gives ordering within a key and parallelism across keys, and both properties depend on the keys being reasonably distributed. When they are not, one partition receives a disproportionate share, its consumer becomes the bottleneck, and adding partitions changes nothing because the hot key still maps to exactly one.

The causes are ordinary. Partitioning orders by merchant when one merchant is forty percent of volume. Partitioning events by country in a business concentrated in one country. A null or default key that a large share of records fall back to.

Detection requires per-partition metrics; the aggregate hides it completely, and the symptom presented to an on-call engineer is "lag is growing on one consumer".

The remedies each cost something. Salting the key — appending a small random suffix — restores distribution and destroys per-key ordering, which is acceptable only if nothing relied on it. Choosing a finer key, such as order identifier rather than merchant, usually works and changes the ordering guarantee to a narrower one. Handling the known hot key on a separate path is ugly and is sometimes the only option.

The decision worth making early is what ordering is genuinely required, because teams routinely partition for an ordering guarantee nothing actually depends on and pay for it in throughput forever.