Consumer lag on a Kafka topic grows during peak and does not recover overnight. You add consumers and nothing improves. Why?

The mechanism

Each partition is assigned to exactly one consumer within a group. With ten partitions and ten consumers, an eleventh consumer joins the group, is assigned nothing, and sits idle consuming resources and contributing nothing.

Parallelism within a consumer group is capped at the partition count. Adding members past that point changes nothing — which is precisely the symptom described.

What to check, in order

1. Partition count versus group size. The immediate confirmation.

2. Per-partition lag, not aggregate. If lag is concentrated on a few partitions, the problem is key skew rather than total capacity — one partition receives disproportionate traffic and its single consumer cannot keep up while others are idle. Adding consumers cannot help that either.

3. Whether the consumer is the bottleneck or something downstream. A consumer waiting on a database is not consuming, and more consumers then add contention rather than throughput.

4. Rebalance frequency. If the group is rebalancing repeatedly, processing stops group-wide each time, and the backlog grows regardless of member count. The signature is consumers being ejected for exceeding the poll interval, which slows everyone, causing more ejections.

The fixes

Increase partitions — the direct fix, and it is awkward: it changes which key lands where, which breaks per-key ordering across the change and invalidates any consumer state keyed by partition. Doable, but plan it rather than doing it during an incident.

Reduce per-message cost — batch downstream writes, remove an N+1 query, move slow work off the poll thread.

Fix skew if lag is concentrated: change the partition key, or add a salt for the hot key and handle the fan-in.

Tune the consumer — reduce max.poll.records so per-poll work fits comfortably inside the poll interval, and enable cooperative rebalancing so unaffected partitions keep processing.

Why the other options are wrong

Retention too short would cause data loss, not lag. Producer batching affects producer efficiency, not consumer parallelism. Auto-commit is a genuine bug source — it can silently skip messages after a crash — but it does not limit throughput.

What a strong answer adds

Noting that partition count is a capacity decision made at design time, because increasing it later is disruptive. Over-provisioning partitions relative to current need is cheap; repartitioning a live topic is not. And that the right alert is oldest-message age rather than lag in messages, since age maps directly to user-visible staleness while a message count does not.