When the queue becomes the outage
How production systems bound asynchronous work queues, apply backpressure, and dig out of backlogs.
The buffer that absorbs your bursts is also the mechanism that sustains your worst outages. This guide reconstructs, from postmortems at GitHub, Honeycomb and incident.io, source code and pull requests at Meta, RabbitMQ and Shopify, and engineering accounts from Slack, Amazon, Netflix, Dropbox and LinkedIn, how real systems answer the one question every queue design must answer: what happens when it is full, and who pays for the drain. A reader leaves able to name their full-queue policy, size recovery capacity, and run a dig-out from a written playbook.
Slack's 2016 job-queue outage wedged because dequeueing a job from full Redis itself required free memory: the full buffer jammed its own exit, and the drain path failed together with the fill path.
What you get out of it
- Every production queue takes one of three stances at its bound (absorb-and-persist, bound-and-refuse, bound-and-drop); most incidents come from operating one stance while having built another.
- The drain path must not depend on the resource that fills: Slack's dequeue needed free Redis memory, and RabbitMQ 4.0 shipped with producer flow control accidentally deleted because its correct behaviour is invisible.
- Kubernetes rejects the newest work while Facebook serves the newest first; both are right, and the flip condition is whether the work's value decays with age.
- Drain rate is surplus capacity only: a consumer fleet autoscaled to steady-state arrivals will never drain a backlog, which is why GitHub expired 200k webhooks under a TTL and Honeycomb had to switch ingest off to bring it back.
- Alert on the age of the oldest unprocessed item, not queue depth: depth is latency (Little's Law), and staleness is the failure users actually see.
Scope
Why this, now. SQS shipped Fair Queues in July 2025 and InfoQ published backlog-drain arithmetic in May 2026; the industry is re-litigating queue bounding and recovery right now, while RabbitMQ 4.0's accidental removal of producer flow control showed how quietly the safety net can vanish.
What it does not cover. Synchronous request-path overload control (retry storms, circuit breakers, metastable request amplification, covered by the 2026-08-29 dig), broker feature comparisons, and exactly-once stream-processing semantics.
Other field guides
Deciding a server is dead
Reconstructs the failure-detection plane behind health checks from six published incidents, five production codebases, four papers and the Kubernetes…
30 sources · 24 organisations · 6 postmortemsThe rule was already written down
A field guide to the machinery between an outage and the next design decision, reconstructed from six first-party incident reports (Cloudflare 2019 a…
34 sources · 24 organisations · 6 postmortemsThe reference architecture Netflix retired
Between 2013 and 2016 the industry copied one company's answer to service-to-service communication: discovery, load balancing, circuit breaking and c…
26 sources · 7 organisations · 4 postmortems