Partition-Bound Concurrency

A partitioned order topic has 24 partitions and a 4-hour backlog. The team scales the consumer group to 120 instances and the backlog does not move. Twenty-four consumers are doing work; the other ninety-six are assigned nothing, because in a partitioned log a partition belongs to exactly one consumer in the group at a time.

Partition-bound concurrency is that ceiling, and it is the single most common surprise in queue capacity planning. It is worth stating as arithmetic: maximum group throughput equals partition count multiplied by the per-consumer rate, and nothing you deploy changes the first term.

Why it matters

It inverts the usual scaling instinct. For a stateless HTTP service, more instances means more throughput; for a partitioned consumer group, more instances past the partition count means slower rebalances and more broker connections for no gain.

The arithmetic also decides whether a backlog is draining at all. At 200 ms of work per message, one consumer thread handles 5 messages a second, so 24 consumers give 120 a second. If events arrive at 170 a second, the backlog grows regardless of instance count, and no amount of scaling will produce a drain time. Teams discover this during an incident, having already spent the scaling option.

And it constrains the fix. Messages already written stay in the partition they landed in, so raising the partition count changes the ceiling for future traffic and does nothing for the existing backlog.

Implementation patterns

• Size partitions from peak throughput divided by per-consumer rate, with headroom: if peak needs 300 consumer-seconds of work per second and one consumer does 5 a second, you need 60 partitions, not 24.
• Reduce per-message cost first. The 200 ms is usually one or two synchronous downstream calls; batching them is the lever with the shortest path to production and it multiplies effective capacity without touching the topic.
• Add intra-consumer concurrency where ordering permits. Process a partition's messages on a thread pool, keyed so that messages sharing a key stay ordered. This raises the per-consumer rate, which is the other term in the product.
• Drain a one-off backlog on a side path. A separate consumer group writing to a differently-partitioned topic, or to object storage for out-of-band processing, is the usual real answer for a large accumulated backlog.
• Alert on age of the oldest unprocessed message, not on count. Count says nothing about whether anyone is waiting; age in seconds is comparable to a product tolerance.

Industry example

The constraint is explicit in Apache Kafka's documented consumer-group model, published with the system's original design papers from 2011 onwards: partitions are the unit of assignment, and consumers beyond the partition count remain idle. It is the same in Kinesis shards and in Pub/Sub's ordered-delivery mode: ordering per key and unlimited consumer parallelism are mutually exclusive, and every ordered system pays this price.

The contrast is instructive. A queue with per-message acknowledgement — SQS, RabbitMQ with a shared queue, or a database-backed job table with row locks — has no such ceiling, and consumers can scale arbitrarily. What it gives up is ordering. Knowing which of the two shapes you have is the first question in any backlog conversation.

Failure scenarios

• Scaling to no effect during an incident, with the extra instances slowing every rebalance.
• A backlog that grows while the status update says it is draining, because the arrival rate was left out of the arithmetic.
• Raising partitions mid-incident, which breaks key-to-partition mapping and therefore per-key ordering across the change, usually discovered later as out-of-order processing.
• Hot partitions. A skewed key — one large tenant, one popular item — means one partition holds a disproportionate share, so the effective ceiling is lower than the partition count suggests.
• Over-partitioning as the reflex fix, where each partition costs broker memory, file handles and replication traffic, and rebalances over several hundred partitions become slow enough to matter.

Trade-offs

When not to use it

The ceiling is not a problem to be removed when ordering per key is the reason the log was chosen. If the consumer applies state transitions that must happen in sequence for an entity, the partition is doing its job and the right lever is per-message cost, never more consumers.

It also does not apply to a queue with per-message acknowledgement, and treating such a queue as if it had a partition ceiling leads to under-provisioning. Check the delivery model before reaching for this reasoning.

Interview question

Q: A partitioned topic with 24 partitions has an 8-hour backlog that is still growing. You have one hour before an executive update. What do you measure, what do you change, and what do you tell them about when it clears?

What a strong answer covers: computing the arrival rate and the per-consumer rate first, because without both there is no drain time to report · establishing whether throughput is below the arrival rate, in which case the honest answer is "it is not draining" · the three levers in order of speed: reduce per-message cost by batching downstream calls, add keyed intra-consumer concurrency, drain to a side path · explaining why adding instances past 24 does nothing and slows rebalances · declining to raise the partition count mid-incident because it breaks per-key ordering and does not touch existing messages · and the signal to report against, which is age of the oldest unprocessed message rather than count.

Quick check

Quiz: What is the maximum throughput of a consumer group on a 24-partition topic where each message costs 200 ms? — About 120 messages a second: 24 partitions × 5 messages per consumer-second. Extra instances are idle.

Flashcard: Which lever helps an existing backlog and which only helps future load? — Reducing per-message cost and adding keyed concurrency help now; raising the partition count only raises the ceiling for messages not yet written.