When the queue becomes the outage: a practitioner field guide

The territory

A system that accepts work faster than it finishes it has to put the difference somewhere. Every design in this guide is an answer to one question: where does the difference go, and what happens when that place is full?

~1T

Items per day through Meta's FOQS priority queue, 2021

Meta Engineering, 2021

33k/s

Peak enqueue rate on Slack's job queue at 1.4B jobs a day, 2017

Slack Engineering, 2017

5M+

Webhook events queued during GitHub's 24-hour degradation, Oct 2018

GitHub post-incident analysis, 2018

73.5h

Longest backlog-sustained outage in the OSDI '22 metastable failure corpus

Huang et al., OSDI 2022

Strip the product names away and the problem is old: arrivals are bursty, service is steady, and a buffer absorbs the difference. The engineering question is not whether to buffer. It is what the buffer promises. A queue that promises delivery (Slack's job queue, Meta's FOQS, Amazon SQS, Dropbox's ATF) is a durability system that happens to smooth load. A queue that promises freshness (a request queue in front of a thread pool, a NATS subscriber buffer) is a latency device that must throw work away to stay honest. Most queue incidents in the public record happen because a team built one kind and operated it as the other.

The surprise in this corpus, and the reason this guide exists, sits in Slack's account of its 2016 job-queue outage: when the Redis cluster backing the queue reached its configured memory limit, the system could no longer enqueue jobs, which everyone expects, and it also could no longer dequeue them, because dequeueing a job required a small amount of free memory to move the job into a processing list. The full buffer jammed its own exit. Draining required, in Slack's words, extensive manual intervention. A queue is supposed to be the thing that saves you during overload; at the exact moment it mattered, it was the thing that needed saving.

"All they're doing is creating a bigger buffer to accumulate data that is in-flight, only to lose it sooner or later. You're making failures more rare, but you're making their magnitude worse." Fred Hebert, Queues Don't Fix Overload, 2014

Hebert's 2014 essay is the sharpest statement of the pessimist position: a queue in front of a persistently overloaded system only changes the shape of the failure, and the real choices are backpressure (make the producer wait) or load shedding (throw work away). The production record collected here largely agrees with him about steady-state overload and complicates the picture for everything else, because the accounts from Slack, Meta and Amazon show durable queues absorbing failures that would otherwise have reached users. The honest synthesis is that a queue moves the decision, it does not remove it. Someone still decides what happens at the bound; the queue just decides whether that decision is made calmly in a design review or at 3 a.m. by whoever is on call.

Figure 1 · Three stances toward the full queue

Every production system in this corpus takes one of three stances when its buffer reaches its bound; the stance, not the broker, is the architectural decision. Sources: Slack, Meta, Reactive Streams, KEP-1040, NATS docs, Fail at Scale.

Diagram source

Scope

This guide covers asynchronous work systems: job queues, task frameworks, message backlogs, and the recovery from them. It deliberately does not cover the synchronous request path's overload loop (retry storms, circuit breakers, metastable request amplification), which this practice has already dug through in retry storms and metastable failure; nor does it compare brokers feature-by-feature or cover exactly-once stream processing semantics.

How it is actually built

Slack, Meta, Dropbox and Amazon each described their asynchronous work system in enough detail to overlay them. The overlap is the reference architecture; the differences are the decisions in section 03.

Figure 2 · The common shape of a production work queue

The durability boundary sits before the scheduler, and failure handling is a first-class path, not an afterthought. Reconstructed from Slack's job queue, Meta's FOQS and Dropbox's ATF.

Diagram source

Three organisations arrived at this shape from three different starting points. Slack began with Redis as both buffer and scheduler, and after the 2016 outage inserted Kafka in front of Redis as a durable buffer, with two new Go services moving jobs in and out, keeping the application-facing queue logic untouched (Slack Engineering, 2017). Meta built FOQS directly on sharded MySQL, and by 2021 it moved roughly one trillion items a day with priorities, per-item delivery delays, and consumer leases with ack/nack semantics (Meta Engineering, 2021). Dropbox's ATF is a callback-based task framework designed for 10,000 tasks per second with at-least-once execution and a stated SLO that 95% of tasks begin within five seconds of their scheduled time (Dropbox, 2020). Nobody in this corpus who described a company-wide work system built it on an in-memory store alone, and two of the three (Slack, Meta) built it explicitly on the database technology their operators already knew how to run.

The durability boundary

The point past which accepted work survives a process death. Slack put it in Kafka, Meta in MySQL, Amazon sells it as SQS. Everything before this boundary is allowed to say no; nothing after it is allowed to lose work silently.

Runs this way at Slack, Meta, Dropbox

Leases, not handoffs

Workers do not own work; they lease it against a deadline. FOQS redelivers an item if the consumer neither acks nor nacks in time, per the customer's retry policy. Amazon's account adds the operational wrinkle: long-running work must heartbeat, or an overloaded worker's lease expires and the item is delivered twice.

Per FOQS and Yanacek, 2019

The sideline path

A dead-letter or surge queue for work the main path cannot finish. Yanacek describes moving excess load into a surge queue with message delay rather than letting it poison the main queue's latency. The sideline is also where poison messages go before they crash a fourth consumer.

Per Yanacek, 2019; motivation at incident.io, 2022

Fairness in front of the store

Multi-tenant queues throttle per tenant before the shared store, so one customer's burst cannot starve the rest. Amazon describes fairness throttling explicitly; SQS productised it as Fair Queues in July 2025; Kubernetes' API server assigns flows to queues by shuffle sharding for the same reason.

Per Yanacek, SQS pricing, KEP-1040

A flow-control layer above compute

Meta's 2023 overview separates the queue from a flow-control layer that meters dequeue into the worker fleet: cross-region balancing, quota management, downstream protection, backoff. The queue holds work; a different component decides how fast to release it. Conflating the two is how backlogs turn into downstream outages.

Per Meta, 2023

Checkpoints inside long jobs

Shopify's job-iteration library makes every long job interruptible: on shutdown a checkpoint is persisted after the current iteration and the job re-enqueues itself. The README's premise is blunt: with frequent deploys, an uninterruptible job "will be either lost or restarted from the beginning." In production since May 2017.

Source: Shopify/job-iteration

One divergence point deserves its own paragraph because it is invisible until it fails: where producer flow control lives. RabbitMQ implements credit flow between the publishing channel and the queue process, so a queue that cannot keep up slows its publishers. In RabbitMQ 4.0 that mechanism was, in the words of the restoring pull request, "unintentionally removed" for classic queues during an unrelated refactor, and the loss shipped. It was found in December 2024 by a user whose publishers ran unthrottled, and restored and backported in PR #12906. A safety mechanism whose normal behaviour is indistinguishable from nothing happening is a mechanism a refactor can delete without failing a single test. If your system relies on backpressure, something in your test suite should fail when it disappears.

The decisions that matter

Four forks in the road, each with a documented production answer on both sides, and the condition that flips each one.

Decision 1 · When the buffer is full, who suffers: producer, oldest work, or newest work?

The three chosen answers

Kubernetes' API server rejects the newest arrival: "an investment in holding a request in a queue has a chance of eventually getting useful work done" (KEP-1040).
Facebook serves the newest first and expires the oldest: adaptive LIFO plus CoDel (Fail at Scale, 2015).
GitHub dropped ~200,000 webhook payloads that outlived an internal TTL rather than deliver them a day late (2018).

Why they disagree

KEP-1040 queues requests whose caller is still waiting and will retry anyway; evicting a queued request wastes the wait already invested.
Facebook's queues front interactive traffic where, during a long queue, "the first-in request has often been sitting around for so long the user may have aborted the action". Serving it first spends capacity on a ghost.
GitHub's webhooks have no waiting caller, but their value decays: a day-old push event is noise to the receiver.

Flips when

Value decays with age (interactive callers, notifications, telemetry): favour the newest; drop or expire the oldest.
Value does not decay (billing, provisioning, compliance): reject the newest, hold the oldest, and push backpressure to producers who can hold state.
If you cannot say which of these your workload is, you have found a requirements gap, not a tuning problem.

Decision 2 · Bound the buffer, or make it durable and effectively unbounded?

Chosen

Slack, after 2016: durable and deep. Kafka in front of Redis so that a consumer slowdown becomes disk usage, not memory exhaustion (2017).
NATS core, the opposite pole: a fixed 65,536-message buffer per subscriber, and the server disconnects slow consumers to protect the whole (docs).

Rejected

Slack rejected simply raising Redis memory: at 33k jobs/s a bigger buffer buys minutes, and the 2016 incident showed the full buffer blocks its own drain.
The Reactive Streams working group rejected unbounded mediating queues entirely; the spec exists "to allow the queues which mediate between threads to be bounded" (spec, v1.0.4).

Flips when

The queue is a delivery contract and producers cannot hold state (webhooks in, client events): durability wins; you will drain later.
The queue is internal plumbing between components you control: bound it and propagate demand; Hebert's essay and the Reactive Streams spec are both arguments that an unbounded internal queue is a deferred out-of-memory error.

Decision 3 · Do you drop from the head by deadline, or shed by priority class?

Chosen

Facebook: drop by delay. Their CoDel variant sets M=5ms and N=100ms and, in the overloaded regime, sloughs off requests whose queueing delay exceeds twice the target (folly/Codel.cpp).
Netflix: shed by class. Requests are pre-tagged non-critical, degraded, or critical, and a moving threshold drops the lowest class first (2020, 2024).
Meta's async platform: shed by delay tolerance; overload defers jobs that declared they can wait (2023).

Rejected

The textbook CoDel discipline itself: folly's comment says dropping at an increasing rate, as the paper prescribes for TCP flows, "empirically works better" replaced with a hard slough-off threshold for RPC.
Uniform shedding: Netflix's 2024 numbers show why; during a 12x prefetch spike, uniform dropping would have taken user-initiated requests down with the prefetches, and priority shedding kept them above 99.4% availability.

Flips when

You have no request taxonomy: delay-based dropping (CoDel-style) works with zero classification effort and protects latency immediately.
You have, or can build, a taxonomy: class-based shedding preserves more value per unit of shed load. The real cost is the taxonomy, which has to exist before the incident.

Decision 4 · Is consumer lag a scaling signal?

Chosen

Netflix's async ingest team scaled consumers on lag, accepting lag during spikes as the point of the design: the durable queue absorbs backpressure that previously propagated "to the edge and clients" (Podila, QCon 2021).

Caveat from the same team

"While lag initially seemed like a good metric to scale on ... you cannot scale down easily": zero lag says nothing about how few workers would still produce zero lag, so lag-based autoscaling ratchets up.

Flips when

Scale up on lag or queue age, scale down on worker utilisation. The 2026 InfoQ backlog-math treatment makes the underlying rule explicit: drain rate is surplus capacity, so a fleet sized for steady state "will never drain a backlog without intervention" (InfoQ, 2026).

Figure 3 · What to do when the queue is at its bound

The tree the four decisions collapse into. Terminal nodes are actions taken by real systems: TTL expiry at GitHub, adaptive LIFO at Facebook, reject-newest in Kubernetes, surge queues at Amazon.

Diagram source

Decision	Chosen	Rejected	Because	Evidence
Full-queue victim	Context-dependent: newest (K8s), oldest (Facebook, GitHub)	A single universal answer	Depends on whether work value decays with age	KEP-1040, Fail at Scale
Buffer bound	Durable-deep for contracts (Slack); hard-bounded for plumbing (NATS, Reactive Streams)	Bigger in-memory buffer	A full in-memory buffer blocked its own drain in 2016	Slack, RS spec
Shedding basis	Delay-based with no taxonomy; class-based with one	Uniform shedding; textbook CoDel drop schedule	Priority shedding held user traffic at 99.4% in a 12x spike; folly found the paper's schedule worse for RPC	Netflix 2024, folly
Autoscaling signal	Lag/age up, utilisation down	Lag alone	Zero lag carries no scale-down information	Podila 2021
Producer flow control	Credit flow / demand signalling, tested for presence	Trusting it implicitly	RabbitMQ 4.0 shipped with it accidentally deleted	PR #12906

One argument from inside a maintainer team is worth reproducing rather than summarising. In 2019 a RabbitMQ engineer proposed entering the flow state earlier for quorum queues, protecting the pessimistic case where publishers outrun consumers. A fellow maintainer pushed back: "This optimizes for the pessimistic case. Should we leave things as is until there's more evidence this workload is common in the wild?" The PR was closed unmerged (#2129), superseded by later flow-control work. The disagreement is the durable lesson: flow control tuned for overload taxes the common case, and the team that owns the queue will, quite reasonably, refuse to pay that tax without field evidence. Your overload posture will not come tuned out of the box; the defaults serve the vendor's median customer.

What broke in production

Four documented incidents and one shipped near-miss. They group into three classes, named here because the sources each name only their own instance: the self-jamming buffer, the dig-out, and the deferred failure.

Figure 4 · The self-jamming buffer: Slack, 2016

The failure sequence reconstructed from Slack Engineering's 2017 account. The step to notice is the last one: dequeue needed the same resource the backlog had exhausted.

Diagram source

Blog account

Slack: the queue that could not empty itself

AssumptionRedis is the buffer; if it fills, we stop accepting new jobs and drain.

What happenedDatabase contention slowed job execution; enqueues continued; Redis hit its configured memory limit. Dequeueing required free memory to move a job into a processing list, so the drain path failed along with the fill path.

Blast radiusAll Slack operations depending on the job queue failed; the queue stayed wedged after the original database contention was resolved; recovery took "extensive manual intervention" (2016).

FixKafka inserted in front of Redis as a durable buffer, decoupling acceptance from scheduling; enqueue and relay moved to dedicated Go services.

Design ruleThe drain path must not depend on the resource that fills. Test dequeue, ack and DLQ writes against a full queue, not an empty one.

SourceScaling Slack's Job Queue, 2017

Source / near-miss

RabbitMQ 4.0: the safety net that vanished silently

AssumptionProducer flow control is part of the broker's identity; no release would remove it.

What happenedThe credit-flow mechanism between publishing channels and classic queue processes was "unintentionally removed in 4.0" during the removal of mirroring. Publishers could outrun queues without being slowed. Found by a user in December 2024, months after release.

Blast radiusNo published outage; unbounded queue growth reproduced by the reporter. The exposure window was every 4.0 deployment with fast publishers on classic queues.

FixFlow control restored behind a flag defaulting to the 3.x behaviour, merged and backported to 4.0.x.

Design ruleBackpressure that works is invisible, and invisible mechanisms get deleted. Keep a test that fails when producers are no longer slowed by a saturated queue.

Sourcerabbitmq-server PR #12906, 2024

Postmortem

GitHub: 24 hours of arrivals, drained under a TTL

AssumptionPausing async delivery during a database incident is free; the backlog can simply be replayed afterwards.

What happenedDuring the October 2018 MySQL partition event, GitHub paused webhooks and Pages builds for data-integrity reasons. Over five million hook events and 80 thousand Pages builds queued. The replay itself then had to be paced against "potentially overloading ecosystem partners with notifications."

Blast radius24 hours of degraded service; roughly 200,000 webhook payloads outlived an internal TTL and were dropped rather than delivered.

FixStatus stayed red until the backlog cleared, an explicit choice of integrity over a shorter incident window; the TTL converted an unbounded replay into a bounded one.

Design ruleA backlog is a load test you did not schedule, aimed at your consumers and at other people's servers. Decide the TTL and the replay rate before the incident, because you will certainly apply one during it.

SourceOctober 21 post-incident analysis, 2018

Postmortem

Honeycomb: to come back up, ingest first had to go down

AssumptionIf ingest degrades, restoring the impaired dependency restores ingest.

What happenedOn 25 July 2023, after roughly ten minutes of partially degraded ingestion, a rapid failure cascade took down most services. Bringing ingest straight back would have re-overloaded the still-cold caches and saturated database connections, so recovery required deliberately circuit-breaking ingest traffic, warming the caches, then reopening.

Blast radiusUser-facing impact 13:40 to 14:48 UTC; ingestion restored around 15:15 and full service at 15:35; described by Honeycomb as their biggest outage since having paying customers.

FixDocumented circuit-breaking as a recovery tool, plus cache-dependency hardening so ingest can start against a cold backend.

Design ruleRecovery load is arrival load plus backlog drain plus cold caches. If you cannot deliberately hold traffic off, the system will re-enter the failed state each time you try to return; an intentional off-switch for intake is a recovery feature.

SourceWhat Comes Up Must First Go Down, 2023

Postmortem

incident.io: one poisoned consumer, whole-app crash loop

AssumptionA failing async consumer fails alone; the deployable around it stays up.

What happenedIn the Google Cloud Pub/Sub client, goroutines started by Receive could panic outside any recover; one bad code path in an event consumer repeatedly crashed the entire application, which served the web product as well. The first hypothesis chased was a poison-pill message that re-triggered the crash on each redelivery.

Blast radiusIntermittent full-product downtime on 30 November 2022 across repeated crash-restart cycles; an engineer was assigned to trigger manual restarts while diagnosis continued.

FixPanic recovery wrapped around consumer goroutines; deploy-time mitigation the same afternoon.

Design ruleRedelivery turns one bad message into a crash loop: the queue faithfully returns the input that kills you. Poison-pill handling (catch, count, sideline) is part of the consumer contract, not defensive polish.

SourceIntermittent downtime from repeated crashes, 2022

The three classes travel well. The self-jamming buffer (Slack, and RabbitMQ's near-miss as its inverse) is any design where admission, storage and drain share a resource or a mechanism, so saturation disables the cure. The dig-out (GitHub, Honeycomb, and the whole OSDI '22 metastable corpus, where backlogs and retries sustain an outage after its trigger is gone) is the recovery phase treated as an afterthought: the OSDI study found retry-driven load amplification in more than half of the incidents it examined, and four of fifteen major AWS outages in the preceding decade were of this shape. The deferred failure (incident.io, and Sidekiq issue #5282's measured one-to-one coupling of queue depth and latency) is the quiet class: an async system converts errors and slowness into backlog, and backlog into staleness, and staleness is invisible unless you alert on the age of the oldest unprocessed item rather than on depth.

Figure 5 · The dig-out, as practised by teams that have done it

A backlog recovery sequence assembled from Honeycomb's circuit-breaking, GitHub's TTL and paced replay, Meta's deferral by delay tolerance and the drain arithmetic. No single source prescribes the whole sequence; the assembly is this guide's.

Diagram source

The absence worth knowing about

No postmortem in this corpus attributes an outage to work legitimately dropped by a bounded queue: the NATS-style pole of the design space has no published disaster stories here, while the durable-unbounded pole has several. Read that carefully rather than triumphantly. It may mean bounded-and-dropping designs fail more safely; it may equally mean their failures (silently lost messages) do not produce postmortems because nobody notices. Both readings argue for the same control: an end-to-end reconciliation that counts what went in against what came out.

Numbers you can plan against

Scale points, control parameters, defaults and prices, each dated. Measured, claimed and derived figures are marked as such.

Metric	Value	At	Kind	As of	Source
Queue throughput, company-wide	~1T items/day	Meta FOQS	Reported	2021	Meta
Job queue peak enqueue	33,000/s (1.4B/day)	Slack	Reported	2017	Slack
Kafka messages	7T/day, 7M partitions	LinkedIn	Reported	2019	LinkedIn
Task framework design target	10,000 tasks/s	Dropbox ATF	Reported	2020	Dropbox
Task start SLO	95% within 5 s	Dropbox ATF	Reported	2020	Dropbox
CoDel-on-RPC parameters	target 5 ms, window 100 ms, slough at 2× target	Facebook (folly)	Reported, in code	2015; code current 2026	folly
Availability held by priority shedding	>99.4% during 12× spike	Netflix	Reported	2024	Netflix
Backlog at pause	5M+ webhooks, 80k builds; ~200k dropped past TTL	GitHub	Reported	2018	GitHub
Slow-consumer defaults	65,536 msgs / ~64 MB / 2 s write deadline	NATS	Vendor default	docs, 2026	NATS
Rented queue price	$0.40/M requests (tiers to $0.24); Fair Queues +$0.10/M	Amazon SQS standard	Vendor price, third-party documented	2025	AWS Fundamentals
Backlog-sustained outage durations	1.5 to 73.53 h	OSDI '22 corpus (AWS, Google, Azure, others)	Measured across 22 incidents	2022	Huang et al.
Drain time	backlog ÷ (capacity − arrivals)	General result	Derived, see below	2026	InfoQ

The drain formula deserves a worked example because its consequences are brutal and routinely ignored. Take GitHub's five million queued webhooks. A delivery fleet with 1,000 events per second of surplus capacity (beyond keeping up with live arrivals) drains it in 5,000,000 / 1,000 ≈ 83 minutes; with 100 events per second of surplus, it takes almost 14 hours, and with zero surplus it never finishes. That last case is not hypothetical: a fleet autoscaled to match steady-state arrivals has, by construction, zero surplus, which is the InfoQ article's point that such systems "will never drain a backlog without intervention" (2026). Surplus for dig-outs is something you buy, borrow from lower-priority work, or create by shedding; it does not appear on its own. Meta creates it by deferring jobs with long delay tolerance; Honeycomb created it by turning intake off entirely.

Read these carefully

The Meta, Slack, LinkedIn and Dropbox scale figures are self-reported by the operating teams in engineering posts and are five to nine years old in some cases; treat them as order-of-magnitude anchors, not current facts. The SQS price is a vendor list price as documented by a third-party guide in 2025 and excludes payload size multiples (each 64 KB chunk bills as one request) and data transfer. The 83-minute figure above is this guide's arithmetic, not GitHub's; their actual drain was paced deliberately and took longer.

The evidence wall

Every source behind this page, graded. Filter by kind. The ledger with per-claim quotes ships alongside this file as sources.md.

Postmortem GitHub2018-10

October 21 post-incident analysis

The canonical published backlog: 5M+ webhook events and 80k Pages builds queued during a 24-hour database incident, drained under a TTL that dropped ~200k payloads, paced to avoid overloading receiving third parties.

Carry forwardDecide TTL and replay pacing before the incident; a backlog is a load test aimed at your consumers and other people's servers.

github.blog/2018-10-30-oct21-post-incident-analysis

Postmortem Honeycomb2023-08

Incident Review: What Comes Up Must First Go Down

An ingest death spiral on 2023-07-25: ten minutes of degraded ingestion, then a cascade. Recovery required deliberately circuit-breaking intake because restarting against cold caches would re-enter the failure.

Carry forwardAn intentional off-switch for intake is a recovery feature; without it every restart replays the overload.

honeycomb.io/blog/incident-review-what-comes-up-must-first-go-down

Postmortem incident.io2022-11

Intermittent downtime from repeated crashes

A panic in a Pub/Sub consumer goroutine, uncatchable by the parent's recover, crash-looped the whole application. The team's first hypothesis was a poison-pill message re-triggering on each redelivery.

Carry forwardRedelivery weaponises bad input; poison-pill catch-count-sideline is part of the consumer contract.

incident.io/blog/intermittent-downtime

Eng blog Slack2017-12

Scaling Slack's Job Queue

1.4B jobs/day, 33k/s peak, and the 2016 outage in which a full Redis could not dequeue because dequeueing needed free memory. The fix: Kafka as a durable buffer in front, with dedicated relay services.

Carry forwardThe drain path must not depend on the resource that fills; separate acceptance from scheduling.

slack.engineering/scaling-slacks-job-queue

Eng blog Amazon (D. Yanacek)2019-12

Avoiding insurmountable queue backlogs

The Builders' Library treatment: backpressure upstream, delay and surge queues, per-tenant fairness throttling, heartbeating long-running work so leases survive overload.

Carry forwardMulti-tenant queues need fairness before the shared store; one tenant's burst is every tenant's backlog otherwise.

aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs

Eng blog Meta2021-02

FOQS: Scaling a distributed priority queue

A priority queue on sharded MySQL moving roughly one trillion items a day, with per-item priority and deliver-after, prefetch buffering, and lease-based ack/nack with customer-defined retry policy.

Carry forwardDurable queues at any scale are databases with a scheduling API; build on the storage your operators already run.

engineering.fb.com/2021/02/22/production-engineering/foqs-scaling-a-distributed-priority-queue

Eng blog Meta2023-01

Asynchronous computing at Meta: Overview and learnings

Introduces delay tolerance as the scheduling primitive: under overload the platform defers jobs that declared they can wait, spreading load over time. A separate flow-control layer meters dequeue with quotas and downstream protection.

Carry forwardAsk every producer for a delay tolerance at enqueue time; it is the cheapest load-shedding taxonomy you will ever build.

engineering.fb.com/2023/01/31/production-engineering/meta-asynchronous-computing

Eng blog Dropbox2020-11

How we designed Dropbox ATF

A company-wide async task framework: 10,000 tasks/s design target, at-least-once execution, no concurrent execution of the same task, and a stated SLO of 95% of tasks starting within 5 seconds of schedule.

Carry forwardPublish start-latency SLOs for async work; without one, "it's queued" silently becomes "it's lost".

dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox

Eng blog Netflix2020-11

Keeping Netflix Reliable Using Prioritized Load Shedding

Requests pre-classified as non-critical, degraded, or critical; a threshold moves with CPU, failure rate and latency, shedding the lowest class first, validated continuously with chaos experiments.

Carry forwardShedding quality is decided by the taxonomy you built before the incident, not by the algorithm during it.

netflixtechblog.com/keeping-netflix-reliable-using-prioritized-load-shedding

Eng blog Netflix2024-06

Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding

The 2020 gateway mechanism pushed into individual services as a library, shedding by CPU and priority bucket. Measured: user-initiated requests at 100% availability while prefetch is throttled; above 99.4% through a 12x prefetch spike.

Carry forwardPriority shedding measurably converts "everyone suffers" into "background work suffers"; the numbers justify the tagging work.

netflixtechblog.com/enhancing-netflix-reliability-with-service-level-prioritized-load-shedding

Eng blog LinkedIn2019-10

How LinkedIn customizes Apache Kafka for 7 trillion messages per day

The published ceiling of queueing scale: 100 clusters, 4,000+ brokers, 100k topics, 7M partitions, 7T messages a day, run on a patched internal Kafka release branch.

Carry forwardAt the top end, operating the queue becomes its own engineering organisation; budget for that or rent.

engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages

Eng blog Fred Hebert2014

Queues Don't Fix Overload

The essay every queue design review should quote: a buffer in front of steady overload accumulates in-flight data "only to lose it sooner or later", making failures rarer but bigger. The real choices are backpressure or shedding.

Carry forwardA queue moves the drop-or-block decision; it never removes it. Find the bottleneck the queue is hiding.

ferd.ca/queues-don-t-fix-overload.html

Article InfoQ2026-05

The Mathematics of Backlogs: Capacity Planning for Queue Recovery

Drain time equals backlog over surplus capacity; a fleet provisioned exactly for steady state never drains; a 10% spike harmless at 80% utilisation is catastrophic at 90%; headroom formulas for recovery-time objectives.

Carry forwardSize consumer fleets for a stated recovery-time objective, not for steady-state throughput.

infoq.com/articles/capacity-planning-queue-recovery

Paper Facebook (B. Maurer)2015-11

Fail at Scale (ACM Queue 13:8)

The queueing chapter of Facebook's reliability practice: FIFO under overload spends capacity on abandoned requests; adaptive LIFO serves the newest during congestion; CoDel with M=5ms, N=100ms bounds standing queues. Shipped in HHVM and Wangle.

Carry forwardQueue discipline is a run-time decision; the right order under load is not the right order at rest.

queue.acm.org/detail.cfm?id=2839461

Paper Nichols & Jacobson2012-07

Controlling Queue Delay (CACM 55:7)

The bufferbloat paper that named the distinction this whole field turns on: good queues convert bursty arrivals into smooth departures; bad queues are standing queues that only add delay. CoDel controls on experienced delay, parameterlessly.

Carry forwardMeasure queues in time waited, not items held; depth is meaningless without the drain rate.

dl.acm.org/doi/10.1145/2209249.2209264

Paper Huang et al., 11 orgs2022

Metastable Failures in the Wild (OSDI '22)

22 incidents across AWS, Google, Azure, IBM, Spotify and others in which a degraded state persisted after its trigger vanished; retry amplification sustained more than half; durations 1.5 to 73.53 hours.

Carry forwardA backlog is a sustaining effect: plan the dig-out as its own failure mode with its own capacity.

usenix.org/conference/osdi22/presentation/huang-lexiang

Decision record Kuberneteschecked 2026-09

KEP-1040: Priority and Fairness for API Server Requests

A complete written argument for bounded per-flow queues with shuffle sharding (a light flow shares all queues with a heavy one at odds around 1 in 5.4 billion), rejection of new arrivals over eviction of queued ones, and an explicit goal order: overload protection, then fairness, then throughput.

Carry forwardWrite the goal order down; every queueing parameter fight is really a fight about it.

github.com/kubernetes/enhancements · keps/1040-priority-and-fairness

Specification Reactive Streams WGv1.0.4

Reactive Streams for the JVM

The industry's agreed protocol for demand: consumers signal request(n), producers may not exceed outstanding demand, and therefore every mediating queue can be bounded. Authored by engineers from Netflix, Lightbend, Pivotal, Red Hat, Twitter and others; absorbed into java.util.concurrent.Flow.

Carry forwardBackpressure is a protocol between components, not a buffer setting inside one.

github.com/reactive-streams/reactive-streams-jvm

Source Meta (folly)current 2026

folly/executors/Codel.cpp

The production CoDel with its divergence documented in a comment: instead of the paper's escalating drop schedule, requests with queueing delay over twice the target are sloughed off during overload, because that "empirically works better for our services".

Carry forwardExpect to adapt textbook control algorithms; keep the deviation and its reason in a comment where the next reader will look.

github.com/facebook/folly · folly/executors/Codel.cpp

Source / PR RabbitMQ2024-12

PR #12906: Restore credit_flow to classic queues

Producer flow control "unintentionally removed in 4.0" during the mirroring removal; a user hit unbounded growth, a contributor restored it behind a flag defaulting to the 3.x behaviour, merged and backported.

Carry forwardTest for the presence of backpressure, not only its behaviour; a silent safety net can vanish in a refactor.

github.com/rabbitmq/rabbitmq-server/pull/12906

Source / rejected PR RabbitMQ2019-2020

PR #2129: Adjust quorum queue flow control (closed unmerged)

A proposal to enter the flow state earlier, protecting publish-heavy pessimistic cases. Closed after a maintainer objected that it "optimizes for the pessimistic case" without evidence the workload is common; superseded by later work.

Carry forwardDefaults protect the vendor's median customer; your overload posture is your own tuning burden, with benchmarks.

github.com/rabbitmq/rabbitmq-server/pull/2129

Source / issue Sidekiq2022-04

Issue #5282: Latency vs queue length

A practitioner's measurements during a Redis-full incident: an apparently one-to-one correlation between queue depth and reported job latency while draining, persisting even in an isolated test with only workers consuming.

Carry forwardLittle's Law shows up whether invited or not: at fixed drain rate, depth is latency. Alert on age of the oldest item.

github.com/sidekiq/sidekiq/issues/5282

Source Shopifysince 2017-05

Shopify/job-iteration

The library that makes Shopify's long-running jobs interruptible: work is expressed as an enumerator, a checkpoint persists after each iteration, and interruption re-enqueues rather than restarts. Premise stated in the README: with frequent deploys, a long job "will be either lost or restarted from the beginning."

Carry forwardMake long jobs resumable at design time; worker churn is routine, and a backlog of half-done restarts is self-inflicted.

github.com/Shopify/job-iteration

Vendor docs NATS (Synadia)checked 2026-09

NATS documentation: Slow Consumers

The bound-and-drop pole stated as philosophy: NATS "favors the approach of protecting the system as a whole over accommodating a particular consumer". Defaults: 65,536 pending messages or ~64 MB per subscriber; the server disconnects a consumer it cannot flush within a 2-second write deadline.

Carry forwardIf you adopt a drop-by-design transport, the delivery guarantee moves into your application layer; budget for it there.

docs.nats.io/running-a-nats-service/nats_admin/slow_consumers

Pricing AWS SQS (via AWS Fundamentals)2025

SQS pricing, documented

Standard queues: $0.40 per million requests after the first million, tiering to $0.24 above 200B/month; each 64 KB of payload bills as one request; Fair Queues (July 2025) add $0.10 per million when tenant grouping is used.

Carry forwardRented queues price per request, so chatty heartbeats and small messages dominate the bill before throughput does.

awsfundamentals.com/blog/sqs-pricing

Talk Netflix (S. Podila)2021

Microservices to Async Processing Migration at Scale (QCon Plus)

The migration of Netflix's playback-data ingest from synchronous microservices, whose backpressure reached clients, to a durable queue. Transcript on InfoQ; the scale-down caveat about lag-based autoscaling is the part worth the visit.

Carry forwardLag tells you when to scale up and nothing about when to scale down; pair it with utilisation.

infoq.com/presentations/migration-microservices-scale

Talk Zach Tellman2015

Everything Will Flow (Clojure/West)

Queueing theory applied to application design: why unbounded queues make failure rare but total, and why every queue needs an explicit policy. Cited here to its published abstract; the video could not be fetched from this session's network, so no timestamp is given.

Carry forwardEvery in-process channel and thread-pool queue is a queue; the ones nobody configured are the ones that fail first.

youtube.com/watch?v=1bNOO3xxMc0

Build a miniature, then productionise it

Six rungs from an evening's toy to an operable system. The line from reading to skill is crossed at rung four, where you practise the dig-out instead of reading about one.

A bounded queue with visible arithmetic

One producer, one consumer pool, one bounded in-memory queue. Instrument arrivals, service rate, depth and time-in-queue; drive arrival rate past service rate and watch depth and latency move together.

Done when: your dashboard shows Little's Law (depth ≈ rate × wait) holding live. Teaches: depth is latency, which is the whole reason bounds exist.

Three full-queue policies behind one flag

Implement reject-newest, drop-oldest-past-TTL, and block-the-producer as switchable policies on the same queue. Overload it identically under each and record which work was lost, how stale delivered work was, and what producers experienced.

Done when: you can produce the three loss/staleness/pushback profiles from one load run each. Teaches: the full-queue policy is a product decision wearing an engineering costume.

Durability boundary and leases

Replace the in-memory queue with a durable log or table. Add consumer leases with deadlines, ack/nack, redelivery, a max-attempts counter and a dead-letter table. Kill workers mid-job and confirm nothing is lost and nothing runs twice unacknowledged.

Done when: kill -9 on any worker at any moment changes nothing about eventual outcomes except timing. Teaches: at-least-once plus idempotency as a lived contract, not a slogan.

The dig-out drill

Stop consumers for an hour under live producer load, then practise recovery: scale consumers, measure drain rate against the surplus-capacity formula, expire past-TTL items, pace the drain against a rate-limited fake downstream.

Done when: predicted and observed drain times match within 20%, and you have a written runbook. Teaches: recovery capacity is bought or created, never assumed.

Poison and fairness

Inject a message that crashes its consumer and confirm it lands in the dead-letter path after N attempts instead of crash-looping the fleet. Then add a second tenant producing 100x the first's volume and add per-tenant throttling until the small tenant's latency stops depending on the big one's behaviour.

Done when: one poison message costs N executions, not the service; tenant A's p99 is flat while tenant B floods. Teaches: the two failure modes that only appear with real, mixed traffic.

Operate it: age alerts and a shedding taxonomy

Alert on oldest-item age per queue, not depth. Tag every message with a priority or delay tolerance at enqueue time, and shed or defer by tag when utilisation crosses a threshold. Run a game day where the on-call recovers from a four-hour synthetic backlog using only the runbook.

Done when: a stranger to the system completes the game day from documentation alone. Teaches: the taxonomy and the runbook are the production system; the broker was the easy part.

Keep hunting

The queries that surfaced this material, grouped by what they find. The vocabulary is the value: backlog, drain, lease, poison pill, slow consumer, credit flow, delay tolerance.

Postmortems and incidents

"incident review" queue backlog ingest outage
postmortem "queue" "ran out of memory" dequeue
"poison pill" consumer crash loop postmortem
site:github.blog post-incident analysis webhooks backlog

Engineering accounts

"job queue" engineering "billion jobs" scaling
"distributed priority queue" engineering blog MySQL shards
"async task framework" architecture "at-least-once"
"prioritized load shedding" engineering

Source, issues, design records

repo:rabbitmq/rabbitmq-server is:pr is:closed is:unmerged "flow control"
"credit_flow" OR "credit flow" rabbitmq removed 4.0
path:keps "priority and fairness" shuffle sharding queues
codel executor site:github.com "overloaded"

Mechanism and theory

"queues don't fix overload"
"adaptive LIFO" CoDel "fail at scale"
"metastable failures" OSDI retry amplification backlog
backlog drain time "surplus capacity" formula

References

GitHub, October 21 post-incident analysis GitHub Blog, 2018-10-30. Checked 2026-09-10.
Honeycomb, Incident Review: What Comes Up Must First Go Down Honeycomb blog, August 2023 (incident 2023-07-25). Checked 2026-09-10.
incident.io, Intermittent downtime from repeated crashes incident.io blog, 2022-11-30. Checked 2026-09-10.
Slack Engineering, Scaling Slack's Job Queue slack.engineering, December 2017. Checked 2026-09-10.
David Yanacek, Avoiding insurmountable queue backlogs Amazon Builders' Library, December 2019. Checked 2026-09-10.
Meta, FOQS: Scaling a distributed priority queue engineering.fb.com, 2021-02-22. Checked 2026-09-10.
Meta, Asynchronous computing at Meta: Overview and learnings engineering.fb.com, 2023-01-31. Checked 2026-09-10.
Dropbox, How we designed Dropbox ATF: an async task framework dropbox.tech, November 2020. Checked 2026-09-10.
Netflix, Keeping Netflix Reliable Using Prioritized Load Shedding Netflix TechBlog, November 2020. Checked 2026-09-10.
Netflix, Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding Netflix TechBlog, 2024-06-25. Checked 2026-09-10.
Shopify, job-iteration GitHub README; in production since May 2017. Checked 2026-09-10.
LinkedIn, How LinkedIn customizes Apache Kafka for 7 trillion messages per day LinkedIn Engineering, October 2019. Checked 2026-09-10.
Ben Maurer, Fail at Scale ACM Queue 13(8), November 2015. Checked 2026-09-10.
Kathleen Nichols and Van Jacobson, Controlling Queue Delay Communications of the ACM 55(7), July 2012. Checked 2026-09-10.
Lexiang Huang et al., Metastable Failures in the Wild OSDI '22, USENIX, 2022. Checked 2026-09-10.
Kubernetes sig-api-machinery, KEP-1040: Priority and Fairness for API Server Requests kubernetes/enhancements, ongoing. Checked 2026-09-10.
Reactive Streams working group, Reactive Streams for the JVM, v1.0.4 GitHub. Checked 2026-09-10.
Meta, folly/executors/Codel.cpp GitHub, current main branch. Checked 2026-09-10.
RabbitMQ, PR #12906: Restore credit_flow between AMQP 0.9.1 channel/MQTT connection and CQ processes GitHub, opened 2024-12-09, merged via #12907. Checked 2026-09-10.
RabbitMQ, PR #2129: Adjust quorum queue flow control (closed unmerged) GitHub, opened 2019-10-02, closed 2020. Checked 2026-09-10.
Sidekiq, Issue #5282: Latency vs Queue Length GitHub, 2022-04-12. Checked 2026-09-10.
NATS documentation, Slow Consumers docs.nats.io, current. Checked 2026-09-10.
AWS Fundamentals, SQS Pricing: Understanding and Optimizing Costs awsfundamentals.com, current as of 2025. Checked 2026-09-10.
InfoQ, The Mathematics of Backlogs: Capacity Planning for Queue Recovery infoq.com, 2026-05-21. Checked 2026-09-10.
Sharma Podila, Microservices to Async Processing Migration at Scale QCon Plus 2021, transcript on InfoQ. Checked 2026-09-10.
Zach Tellman, Everything Will Flow Clojure/West 2015, video. Abstract checked 2026-09-10; video not fetchable from this session's network.
Fred Hebert, Queues Don't Fix Overload ferd.ca, 2014. Checked 2026-09-10.