Competing Consumers

The standard way to scale asynchronous work, and it is nearly free: add another consumer and throughput increases, with the queue distributing work naturally to whoever is ready.

The properties that make it attractive: automatic load balancing, since a slow consumer simply takes fewer messages; resilience, since a consumer that dies mid-message causes redelivery to another; and elasticity, since consumers can be scaled on queue depth.

The constraints that must be respected. Ordering is lost across the consumer group — messages are processed concurrently, so a message published later can complete first, which breaks anything depending on sequence. Where per-entity ordering matters, partition by key so all messages for one entity go to one consumer, accepting the throughput cap per key that this implies.

At-least-once delivery means duplicates, so consumers must be idempotent; a consumer that crashes after doing the work and before acknowledging will see the message again.

Poison messages block progress unless handled, which is why a dead letter path is part of the pattern rather than an addition.

The scaling limit to know: throughput scales with consumers until the downstream becomes the bottleneck, at which point adding consumers converts a queue backlog into database contention. The queue was providing backpressure, and removing it by over-scaling consumers is a common way to turn a delay into an outage.