Reliability & Operations 10 Sep 2026 31 min read

When the queue becomes the outage

How production systems bound asynchronous work queues, apply backpressure, and dig out of backlogs.

The buffer that absorbs your bursts is also the mechanism that sustains your worst outages. This guide reconstructs, from postmortems at GitHub, Honeycomb and incident.io, source code and pull requests at Meta, RabbitMQ and Shopify, and engineering accounts from Slack, Amazon, Netflix, Dropbox and LinkedIn, how real systems answer the one question every queue design must answer: what happens when it is full, and who pays for the drain. A reader leaves able to name their full-queue policy, size recovery capacity, and run a dig-out from a written playbook.

The finding that surprised me

Slack's 2016 job-queue outage wedged because dequeueing a job from full Redis itself required free memory: the full buffer jammed its own exit, and the drain path failed together with the fill path.

What you get out of it

  • Every production queue takes one of three stances at its bound (absorb-and-persist, bound-and-refuse, bound-and-drop); most incidents come from operating one stance while having built another.
  • The drain path must not depend on the resource that fills: Slack's dequeue needed free Redis memory, and RabbitMQ 4.0 shipped with producer flow control accidentally deleted because its correct behaviour is invisible.
  • Kubernetes rejects the newest work while Facebook serves the newest first; both are right, and the flip condition is whether the work's value decays with age.
  • Drain rate is surplus capacity only: a consumer fleet autoscaled to steady-state arrivals will never drain a backlog, which is why GitHub expired 200k webhooks under a TTL and Honeycomb had to switch ingest off to bring it back.
  • Alert on the age of the oldest unprocessed item, not queue depth: depth is latency (Little's Law), and staleness is the failure users actually see.

Scope

Why this, now. SQS shipped Fair Queues in July 2025 and InfoQ published backlog-drain arithmetic in May 2026; the industry is re-litigating queue bounding and recovery right now, while RabbitMQ 4.0's accidental removal of producer flow control showed how quietly the safety net can vanish.

What it does not cover. Synchronous request-path overload control (retry storms, circuit breakers, metastable request amplification, covered by the 2026-08-29 dig), broker feature comparisons, and exactly-once stream-processing semantics.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.