A queue consumer group processes 500 messages per second and the backlog is growing. The team doubles the consumer count from 6 to 12 and throughput stays at 500 per second. What is the most likely reason?

The mechanism

In a partitioned log, a partition is assigned to exactly one consumer in a group at a time. That rule is what preserves per-partition ordering, and it sets a hard ceiling on parallelism: the maximum number of usefully busy consumers in a group equals the number of partitions. With 6 partitions and 12 consumers, 6 of them are assigned nothing at all. They are running, healthy, reporting as members of the group, and doing no work.

This is the general shape of a parallelism limit, and it is worth recognising outside queues too: adding workers raises throughput only up to the number of independent work streams. Beyond that, the extra workers are cost without effect.

What to do instead

• Add partitions, not consumers. Throughput per partition times partition count is the ceiling. If each consumer sustains about 85 messages per second, reaching 2,000 per second needs roughly 24 partitions and 24 consumers.
• Check what one consumer is actually waiting for before scaling anything. If each message makes a 100 ms database call, one consumer handles about 10 messages per second serially, and the fix is concurrency or batching inside the consumer rather than more processes.
• Partitions are not free to add later. Increasing the count changes the key-to-partition mapping, so messages for one key can land on a new partition while older messages for that key are still unprocessed on the old one, breaking ordering during the transition. Size the partition count for a few years of growth at creation time; a few dozen partitions for a modest topic is normal and cheap.

Why the other options fail

• "The consumers need more memory to batch effectively." Batching does raise per-consumer throughput, and that is why this is a plausible wrong answer. It cannot explain an exactly unchanged 500 per second after doubling the fleet; a memory constraint would show as per-consumer throughput below what one consumer can do, not as added consumers having zero effect.
• "The broker throttles consumer groups above 6 members." Brokers do have quotas, usually per client or per user and expressed in bytes per second, not member counts. A quota would also have been hit before the change.
• "Message ordering forces serial processing across all partitions." Ordering is guaranteed per partition, not across a topic. If it were global, 6 consumers would not have reached 500 per second either.

When not to add partitions

If the backlog is a one-off and drains within the hour, do nothing: a consumer group running at its partition ceiling is operating as designed. The time to act is when the backlog's oldest-message age grows across a full traffic cycle, because age, not depth, is the signal that capacity is structurally short. And if each message is slow because of one downstream call, more partitions just spread the same wait across more processes - fix the call or batch it first.