Message Queue

Definition

A queue holds units of work until a worker takes them. The distinguishing property against an event stream is intent: a queue message is a command addressed to whoever processes it, usually consumed once by one worker, and typically deleted after acknowledgement.

Why it matters

Queues buy three things at once: time decoupling (the producer does not wait), burst absorption (arrival rate may exceed processing rate temporarily), and retryability (a failed unit of work can be attempted again without the user knowing).

They also introduce three obligations most teams under-plan: the user must learn the outcome somehow, the queue can grow without bound, and messages that can never succeed must go somewhere.

Industry example

GitHub's background job system is a good illustration of scale reached by a deliberately unglamorous mechanism. Repository events fan out into large volumes of asynchronous work — webhooks, notifications, search indexing, integrations. None of it may block a git push, because one pathological repository would then degrade pushes for every user.

The instructive detail is queue partitioning by priority and by class of work. A single shared queue means a flood of low-value jobs starves urgent ones, and one slow job type consumes every worker. Separate queues with separate worker pools are a bulkhead: the slow class saturates its own capacity and nothing else. Teams almost always discover this the expensive way, during an incident in which notifications were delayed for hours because someone triggered a bulk operation.

Implementation patterns

• At-least-once delivery with idempotent handlers. Assume every message may be processed twice.
• Visibility timeout / lease. A message being processed is invisible to other workers for a bounded period; if the worker dies, it reappears. The timeout must exceed the realistic processing time or you get duplicate concurrent processing.
• Dead-letter queue. After N attempts, move the message aside with its error, and alert on the DLQ depth. An unmonitored DLQ is a silent data-loss mechanism.
• Exponential backoff with jitter on retry, so failures do not resynchronise into waves.
• Separate queues per priority and per work class, with their own worker pools.
• Bounded queue depth with an explicit policy when full: shed, reject at the edge, or apply backpressure to the producer. Unbounded queues convert a throughput problem into a memory problem, then into an outage.

Failure scenarios

• The poison message. One malformed item fails forever, is retried forever, and consumes the workers. Fixed by a retry limit and a DLQ.
• Queue depth grows without limit during an incident, so recovery takes hours after the fault is fixed — the backlog is now the outage.
• Ordering assumed but not guaranteed. Two workers process "create" and "update" for the same entity concurrently and the update lands first.
• The user never learns the outcome. Work was accepted and silently failed. Every async acceptance needs a status the user or an operator can query.

Trade-offs

Against a synchronous call you gain availability and burst tolerance, and you pay in eventual consistency, a new component to operate, and a much harder debugging story — the stack trace ends at the enqueue.

Against a full event-streaming platform, a database-backed queue is dramatically simpler to operate, has transactional semantics with your own data (which makes the outbox pattern trivial), and is entirely adequate up to a surprisingly high volume. Reach for a broker when you need multiple independent consumers of the same events, replay, or throughput a database cannot serve — not because it is the modern choice.

Interview question

"Your job queue has 4 million messages backed up after a two-hour incident. Walk me through recovery, and tell me what you would change so it cannot happen the same way again."