A partitioned order-events topic has 24 partitions and a 4-hour backlog of 2.4 million messages. Each message takes about 200 ms of single-threaded work. The team wants the backlog cleared within 30 minutes and proposes scaling the consumer group to 120 instances. Roughly what throughput is needed and what will actually happen?

The assumptions, stated

• 2.4 million messages to clear.
• 30 minutes to clear them, and new messages keep arriving at the normal rate — assume the steady rate is roughly 170 per second, which is what produced 2.4 million over 4 hours.
• 200 ms of single-threaded work per message, so one consumer thread processes 5 messages per second.

The arithmetic

Drain requirement: 2,400,000 messages ÷ 1,800 seconds = 1,333 per second, plus the arriving 170 per second, so call it about 1,500 per second.

Consumers needed at 5 messages per second each: 1,500 ÷ 5 = 300 consumer threads.

Available parallelism: 24. In a partitioned log with ordered consumption, a partition is assigned to exactly one consumer in the group at a time. 24 partitions means at most 24 consumers do any work; instances 25 to 120 are assigned nothing and sit idle.

Actual throughput at 24 consumers: 24 × 5 = 120 messages per second, which is below the 170 per second arrival rate. The backlog is not draining at all. It is growing.

Time to clear with 24 consumers: never, at this arrival rate.

Which assumption dominates the error

The 200 ms per message. Halving it doubles throughput, and it is usually the cheapest thing to change — the 200 ms is typically one or two synchronous calls to another service that could be batched. The partition count is the structural limit, but per-message cost is the lever with the shortest path to production.

The arrival rate is the second-most important and the most often forgotten. A backlog plan that ignores incoming traffic computes a drain time for a system that has been switched off.

What the number rules in and out

It rules out horizontal scaling as the fix. It rules in three things, in order of how fast they can be done:

1. Increase per-consumer concurrency without changing partition count — process a partition's messages on a pool of threads where ordering permits, or batch the downstream calls. This is the only option that works today.
2. Raise the partition count. It permits more consumers from now on, and it does not help the existing backlog, because messages already written stay in the partition they landed in. It also changes key-to-partition mapping, so ordering guarantees across the change are broken.
3. Drain to a side path. Read the backlog with a separate consumer group that writes to a differently partitioned topic or to object storage, and process it out of band. This is the usual real answer for a one-off backlog of this size.

When not to reach for more partitions

Raising the partition count is the reflex and it is the wrong first move here, because it does nothing for messages already written and it breaks per-key ordering across the change. It is also not free standing: each partition costs broker memory, open file handles and replication traffic, and a group rebalance over 200 partitions is measurably slower than over 24. Raise partitions to change the ceiling for future load, never to drain a backlog that already exists — and only after confirming the per-message cost cannot be reduced, which is almost always the cheaper lever.

Why the other options fail

120 instances sharing partitions. The common and costly misunderstanding. In a partitioned log, partitions are the unit of assignment and are not shared; the extra instances are warm standbys, not workers. They do consume broker connections and group-coordination overhead, so adding them makes rebalances slower. A queue with per-message acknowledgement — the other main shape of competing consumers — does allow arbitrary consumer counts, and knowing which of the two you have is the whole point of the question.

800 per second and existing consumers suffice. Arithmetic error from ignoring the arrival rate or using an hour instead of 30 minutes. It is the mistake that produces a confident status update and a longer incident.

Cannot estimate without disk speed. The bottleneck here is 200 ms of consumer-side work, not broker I/O; a partitioned log sequentially reading a 4-hour backlog is not disk-bound on any modern hardware. Asking for more data when the binding constraint is already in the stem is how an estimation question is failed.