When the queue becomes the outage
How production systems bound asynchronous work queues, apply backpressure, and dig out of backlogs.
The buffer that absorbs your bursts is also the mechanism that sustains your worst outages. This guide reconstructs, from postmortems at GitHub, Honeycomb and incident.io, source code and pull requests at Meta, RabbitMQ and Shopify, and engineering accounts from Slack, Amazon, Netflix, Dropbox and LinkedIn, how real systems answer the one question every queue design must answer: what happens when it is full, and who pays for the drain. A reader leaves able to name their full-queue policy, size recovery capacity, and run a dig-out from a written playbook.
Slack's 2016 job-queue outage wedged because dequeueing a job from full Redis itself required free memory: the full buffer jammed its own exit, and the drain path failed together with the fill path.
What you get out of it
- Every production queue takes one of three stances at its bound (absorb-and-persist, bound-and-refuse, bound-and-drop); most incidents come from operating one stance while having built another.
- The drain path must not depend on the resource that fills: Slack's dequeue needed free Redis memory, and RabbitMQ 4.0 shipped with producer flow control accidentally deleted because its correct behaviour is invisible.
- Kubernetes rejects the newest work while Facebook serves the newest first; both are right, and the flip condition is whether the work's value decays with age.
- Drain rate is surplus capacity only: a consumer fleet autoscaled to steady-state arrivals will never drain a backlog, which is why GitHub expired 200k webhooks under a TTL and Honeycomb had to switch ingest off to bring it back.
- Alert on the age of the oldest unprocessed item, not queue depth: depth is latency (Little's Law), and staleness is the failure users actually see.
Scope
Why this, now. SQS shipped Fair Queues in July 2025 and InfoQ published backlog-drain arithmetic in May 2026; the industry is re-litigating queue bounding and recovery right now, while RabbitMQ 4.0's accidental removal of producer flow control showed how quietly the safety net can vanish.
What it does not cover. Synchronous request-path overload control (retry storms, circuit breakers, metastable request amplification, covered by the 2026-08-29 dig), broker feature comparisons, and exactly-once stream-processing semantics.
Other field guides
The cluster nobody finished emptying: ten years of GitHub
A single-company dig built from eight GitHub postmortems and availability reports, four design posts that name a rejected option, and first-hand git …
35 sources · 10 organisations · 8 postmortemsWhen the monitor shares the outage
Sorts eleven public incidents into three failure classes (shared fate, silent pipeline, blind by design), draws the three failure domains of a monito…
40 sources · 37 organisations · 11 postmortemsThe delete lands one level up, on the path nobody guarded
Reconstructs the safe-delete architecture from the Atlassian 2022, Azure DevOps 2023, Google Cloud GCVE 2024, Cloudflare R2 2025, AWS S3 2017, GitLab…
34 sources · 21 organisations · 7 postmortems