A rendering platform processes jobs from a queue with a pool of workers. What does the competing-consumers pattern require to be correct, and what limits its scaling?

What it requires to be correct

1. Idempotent processing. A message may be delivered more than once — after a visibility timeout expires, a worker crashes, or an acknowledgement is lost. Without idempotency, a render is produced twice or a charge is applied twice.

2. Visibility timeouts longer than realistic processing time, and extended by long-running work. A timeout shorter than the job means another worker picks it up while the first is still running, which is the most common cause of duplicate processing.

3. A retry ceiling with a dead-letter path. A message that consistently fails must not consume workers forever. Without this, one poisonous job silently reduces effective capacity.

4. Acknowledgement after completion, not on receipt. Acknowledging early means a crash loses the work silently.

5. Bounded concurrency per worker and in aggregate, so the pool does not overwhelm a downstream dependency — the workers are usually not the constraint.

What limits its scaling

Ordering requirements. Competing consumers explicitly give up ordering: several workers process concurrently, so completion order is arbitrary. If order matters, the usual answer is partitioning — ordering guaranteed within a partition key, several partitions processed concurrently — which gives per-entity ordering without a global bottleneck.

Downstream capacity. Adding workers is easy; the constraint is usually a database, an external API's rate limit, or a shared resource. Scaling the pool past that point increases contention and reduces throughput.

Skew. If one tenant's jobs dominate the queue, they occupy all workers. First-in-first-out is not a scheduling policy; fair queueing across tenants is what prevents one customer's bulk export from starving everyone's interactive work.

The diagnostic when the queue grows

Distinguish the causes rather than adding workers: insufficient capacity (high utilisation) · a slow downstream dependency (workers busy waiting, low CPU — adding workers makes it worse) · oversized messages (messages per second falling while bytes per second rises) · a poison message (high redelivery count) · or an ordering constraint meaning one effective consumer regardless of pool size.