Evidence ledger
27 sources
Checked 10 Sep 2026
Evidence ledger
One row per claim in When the queue becomes the outage: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Tier grades follow the implementation-archaeology hierarchy
(postmortem > source > adr > casestudy > paper > talk > blog > vendor).
All sources were located and read through web search and direct fetch in this session on
2026-09-10. This session ran in an egress-restricted environment: GitHub-hosted content was
fetched directly; all other pages were retrieved through the search tool's server-side fetch,
and quotes marked as verbatim are ones the retrieval returned in quotation marks.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitHub | October 21 post-incident analysis | postmortem | 2018-10-30 | 2026-09-10 | https://github.blog/2018-10-30-oct21-post-incident-analysis/ | During the 24 h degradation GitHub queued rather than dropped async work | "over five million hook events and 80 thousand Pages builds queued" |
| 1b | GitHub | (same) | postmortem | 2018-10-30 | 2026-09-10 | https://github.blog/2018-10-30-oct21-post-incident-analysis/ | Part of the backlog was deliberately expired rather than delivered | "processed ~200,000 webhook payloads that had outlived an internal TTL and were dropped" |
| 1c | GitHub | (same) | postmortem | 2018-10-30 | 2026-09-10 | https://github.blog/2018-10-30-oct21-post-incident-analysis/ | Draining had to be throttled to protect third parties | GitHub balanced backlog processing "against potentially overloading ecosystem partners with notifications" |
| 2 | Honeycomb | Incident Review: What Comes Up Must First Go Down | postmortem | 2023-08 (incident 2023-07-25) | 2026-09-10 | https://www.honeycomb.io/blog/incident-review-what-comes-up-must-first-go-down | Recovery from an ingest death spiral required deliberately cutting ingest traffic first | recovery "may require manual circuit-breaking of traffic ... bringing ingest back without a cache would cause it to go down again through overload or database connection saturation" |
| 2b | Honeycomb | (same) | postmortem | 2023-08 | 2026-09-10 | https://www.honeycomb.io/blog/incident-review-what-comes-up-must-first-go-down | Blast radius | user-facing impact 13:40–14:48 UTC on 2023-07-25; ingestion back ~15:15, full restore 15:35; "following 10 minutes of partially degraded ingestion, they saw a rapid failure cascade" |
| 3 | incident.io | Intermittent downtime from repeated crashes | postmortem | 2022-11-30 | 2026-09-10 | https://incident.io/blog/intermittent-downtime | A single bad async consumer crashed the whole app repeatedly; first hypothesis was a poison-pill message | Pub/Sub Receive "goroutines ... could panic without being caught by parent-level recover statements"; initial hypothesis "a bad Pub/Sub event ... that triggers failure cases whenever it is processed" |
| 4 | Slack | Scaling Slack's Job Queue | blog | 2017-12 | 2026-09-10 | https://slack.engineering/scaling-slacks-job-queue/ | Scale of the job queue | "over 1.4 billion jobs" on busiest days, "peak rate of 33,000 per second" |
| 4b | Slack | (same) | blog | 2017-12 | 2026-09-10 | https://slack.engineering/scaling-slacks-job-queue/ | The 2016 outage mechanism: full Redis blocked its own drain | database contention slowed job execution until "Redis reached its maximum configured memory limit"; "the system required free Redis memory in order to dequeue a job", so recovery needed "extensive manual intervention" |
| 4c | Slack | (same) | blog | 2017-12 | 2026-09-10 | https://slack.engineering/scaling-slacks-job-queue/ | The structural fix was durability in front, not a bigger buffer | Kafka was added in front of Redis "as a durable buffer", with two new Go services moving jobs in and out |
| 5 | AWS (David Yanacek) | Avoiding insurmountable queue backlogs, Amazon Builders' Library | blog | 2019-12 | 2026-09-10 | https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/ | Amazon's toolkit: backpressure, delay/surge queues, per-tenant fairness, heartbeats | strategies include "sending backpressure upstream", "using delay queues to put off work until later" via "a surge queue with message delay", "fairness throttling in multi-tenancy systems", "heartbeating long-running messages" |
| 6 | Meta | FOQS: Scaling a distributed priority queue | blog | 2021-02-22 | 2026-09-10 | https://engineering.fb.com/2021/02/22/production-engineering/foqs-scaling-a-distributed-priority-queue/ | Scale and shape: priority queue on sharded MySQL with leases | "processes roughly one trillion items every day"; dequeue leases: consumer gets time to ack/nack, otherwise "FOQS may redeliver the item according to the customer's retry policy" |
| 7 | Meta | Asynchronous computing at Meta: Overview and learnings | blog | 2023-01-31 | 2026-09-10 | https://engineering.fb.com/2023/01/31/production-engineering/meta-asynchronous-computing/ | Delay tolerance as the scheduling primitive; overload handled by time-shifting | "when the service is overloaded, Meta defers jobs with a long delay tolerance so that the workload is spread over a longer time window"; flow-control layer provides "cross-regional load balancing, quota management, rate limiting, downstream protection, backoff and retry" |
| 8 | Dropbox | How we designed Dropbox ATF: an async task framework | blog | 2020-11 | 2026-09-10 | https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox | Scale target and SLOs of a company-wide task framework | designed to "handle 10,000 async tasks per second"; serves "more than 9000 async tasks scheduled per second"; "95% of tasks begin execution within 5 seconds"; at-least-once execution |
| 9 | Netflix | Keeping Netflix Reliable Using Prioritized Load Shedding | blog | 2020-11 | 2026-09-10 | https://netflixtechblog.com/keeping-netflix-reliable-using-prioritized-load-shedding-6cc827b02f94 | Requests are pre-classified into priority buckets so shedding is a lookup, not a decision | requests classified as "non-critical, degraded experience, and critical"; shedding threshold moves as "CPU usage, failure rate, and latency begin to rise" |
| 10 | Netflix | Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding | blog | 2024-06-25 | 2026-09-10 | https://netflixtechblog.com/enhancing-netflix-reliability-with-service-level-prioritized-load-shedding-e735e6ce8f7d | Measured effect of priority shedding | "user-initiated requests maintain 100% availability and only prefetch requests are throttled"; during an infrastructure outage availability of user-initiated requests stayed "above 99.4% despite a 12x spike in prefetch requests" |
| 11 | Shopify | job-iteration README (GitHub) | source | in production since 2017-05, checked 2026-09-10 | 2026-09-10 | https://github.com/Shopify/job-iteration | Long-running jobs must checkpoint because workers are interrupted routinely | "with frequent deploys and worker restarts ... a job will be either lost or restarted from the beginning"; on interruption "a checkpoint will be persisted to Redis after the current each_iteration, and the job will be re-enqueued"; "Software that is designed for high availability must be resilient to interruptions that come from the infrastructure" |
| 12 | How LinkedIn customizes Apache Kafka for 7 trillion messages per day | blog | 2019-10 | 2026-09-10 | https://engineering.linkedin.com/blog/2019/apache-kafka-trillion-messages | Upper bound of published queueing scale | "7 trillion per day", "100 Kafka clusters with more than 4,000 brokers", "7 million partitions" | |
| 13 | Facebook (Ben Maurer) | Fail at Scale | paper | 2015-11 (ACM Queue 13(8)) | 2026-09-10 | https://queue.acm.org/detail.cfm?id=2839461 | FIFO is the wrong discipline during overload | "the first-in request has often been sitting around for so long the user may have aborted the action that generated the request"; under load "the server switches to LIFO mode" |
| 13b | Facebook (Ben Maurer) | (same) | paper | 2015-11 | 2026-09-10 | https://queue.acm.org/detail.cfm?id=2839461 | CoDel-on-queues parameters used at Facebook | "A value of five milliseconds for M and 100ms for N tends to work well across a wide set of use cases"; implementations in HHVM and the open-source Wangle library |
| 14 | Nichols & Jacobson | Controlling Queue Delay | paper | 2012-07 (CACM 55(7)) | 2026-09-10 | https://dl.acm.org/doi/10.1145/2209249.2209264 | The good-queue/bad-queue distinction and delay-based control | good queues "convert bursty arrivals into smooth, steady departures"; bad queues are "standing queues that simply create excess delay"; CoDel keeps minimum delay below 5 ms and "is parameterless" |
| 15 | Huang et al. (11 orgs studied) | Metastable Failures in the Wild | paper | 2022 (OSDI '22) | 2026-09-10 | https://www.usenix.org/conference/osdi22/presentation/huang-lexiang | Backlogs sustain outages after the trigger is gone | retry amplification "present in more than 50% of all incidents studied"; "at least 4 of 15 major AWS outages in the last decade were metastable failures", durations "1.5 to 73.53 hours" |
| 16 | Kubernetes sig-api-machinery | KEP-1040: Priority and Fairness for API Server Requests | adr | ongoing, checked 2026-09-10 | 2026-09-10 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness | When a bounded queue fills, reject the newcomer, keep the queued | rejects arriving requests rather than evicting because "an investment in holding a request in a queue has a chance of eventually getting useful work done" |
| 16b | Kubernetes sig-api-machinery | (same) | adr | checked 2026-09-10 | 2026-09-10 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness | Shuffle sharding isolates heavy flows from light ones | "the probability of a given light flow hashing to the same set of 6 queues as the heavy flow is about one in 5.4 billion"; goal order: "overload protection" first, "fairness" second, "throughput" third |
| 17 | Reactive Streams WG (Kaazing, Lightbend, Netflix, Pivotal, Red Hat, Twitter et al.) | Reactive Streams specification for the JVM, v1.0.4 | adr | v1.0.4, checked 2026-09-10 | 2026-09-10 | https://github.com/reactive-streams/reactive-streams-jvm | Backpressure exists to make mediating queues boundable | "The purpose of Reactive Streams is to provide a standard for asynchronous stream processing with non-blocking backpressure"; "backpressure is an integral part of this model in order to allow the queues which mediate between threads to be bounded" |
| 18 | Meta (folly) | folly/executors/Codel.cpp | source | current main, checked 2026-09-10 | 2026-09-10 | https://github.com/facebook/folly/blob/main/folly/executors/Codel.cpp | Production CoDel deliberately diverges from the paper | code comment: "we slough off requests with queueing delay > 2*target_delay while in the overloaded regime. This empirically works better for our services than the codel approach of increasingly often dropping packets." |
| 19 | RabbitMQ | PR #12906: Restore credit_flow between AMQP 0.9.1 channel/MQTT connection -> CQ processes | source | 2024-12-09, merged via #12907 | 2026-09-10 | https://github.com/rabbitmq/rabbitmq-server/pull/12906 | Producer flow control was removed by accident and nobody noticed until a user hit unbounded growth | "The credit_flow between publishing AMQP 0.9.1 channel (or MQTT connection) and (non-mirrored) classic queue processes was unintentionally removed in 4.0"; fix merged and backported to v4.0.x |
| 20 | RabbitMQ | PR #2129: Adjust quorum queue flow control (closed unmerged) | source | opened 2019-10-02, closed 2020-04 | 2026-09-10 | https://github.com/rabbitmq/rabbitmq-server/pull/2129 | Earlier flow-state entry was rejected for lack of evidence the workload is common | maintainer: "This optimizes for the pessimistic case. Should we leave things as is until there's more evidence this workload is common in the wild?"; closed as superseded |
| 21 | Sidekiq (Dan Mayer, reporter) | Issue #5282: Latency vs Queue Length | source | 2022-04-12 | 2026-09-10 | https://github.com/sidekiq/sidekiq/issues/5282 | Practitioner-measured coupling of backlog depth and job latency during a Redis-full incident | "we had a pretty direct looking 1:1 correlation of queue_size decreasing along with the latency" |
| 22 | NATS (Synadia) | NATS documentation: Slow Consumers | vendor | current, checked 2026-09-10 | 2026-09-10 | https://docs.nats.io/running-a-nats-service/nats_admin/slow_consumers | The opposite design pole: shed the consumer, never buffer unboundedly | "NATS favors the approach of protecting the system as a whole over accommodating a particular consumer"; server will "disconnect the connection with the slow consumer to protect itself"; defaults: 65,536 pending messages / ~64 MB, write_deadline 2 s |
| 23 | AWS Fundamentals (third-party documentation of AWS pricing) | SQS Pricing: Understanding and Optimizing Costs | vendor | current, checked 2026-09-10 | 2026-09-10 | https://awsfundamentals.com/blog/sqs-pricing | Unit economics of rented queueing | standard queues: first 1M requests/month free, then "\(0.40 per million requests up to 100 billion", tiering down to \)0.30 and \(0.24; Fair Queues (July 2025) add \)0.10/M when MessageGroupId is used |
| 24 | InfoQ | The Mathematics of Backlogs: Capacity Planning for Queue Recovery | blog | 2026-05-21 | 2026-09-10 | https://www.infoq.com/articles/capacity-planning-queue-recovery/ | Drain time depends on surplus capacity only | "Backlog drain time depends on surplus capacity (total processing rate minus arrival rate)"; systems "provisioned exactly for steady-state traffic have zero recovery capacity and will never drain a backlog without intervention" |
| 24b | InfoQ | (same) | blog | 2026-05-21 | 2026-09-10 | https://www.infoq.com/articles/capacity-planning-queue-recovery/ | Utilization nonlinearity hides the cliff | "the same 10% traffic spike that is barely noticeable at 80% utilization can be catastrophic at 90%" |
| 25 | Netflix (Sharma Podila) | Microservices to Async Processing Migration at Scale (QCon Plus 2021, InfoQ transcript) | talk | 2021, checked 2026-09-10 | 2026-09-10 | https://www.infoq.com/presentations/migration-microservices-scale/ | A durable queue absorbs backpressure that previously reached clients; lag is a poor scale-down signal | synchronous ingest "could propagate intermittent back pressure to the edge and clients"; on lag-based autoscaling: "while lag initially seemed like a good metric to scale on ... you cannot scale down easily" |
| 26 | Zach Tellman | Everything Will Flow (Clojure/West 2015) | talk | 2015 | 2026-09-10 | https://www.youtube.com/watch?v=1bNOO3xxMc0 | Unbounded queues make failure rare but total; queueing theory belongs in application design | published abstract: naive use of queueing primitives creates "brittle, unpredictable systems that fall over when we least expect". Video not fetchable from this session's network; claim is cited to the talk's published abstract, not to a timestamp. |
| 27 | Fred Hebert | Queues Don't Fix Overload | blog | 2014 | 2026-09-10 | https://ferd.ca/queues-don-t-fix-overload.html | A queue in front of overload only converts frequent small failures into rare huge ones | "all they're doing is creating a bigger buffer to accumulate data that is in-flight, only to lose it sooner or later"; "you're making failures more rare, but you're making their magnitude worse"; the real options are backpressure or load shedding |
Notes on evidence quality
- The three postmortems (GitHub, Honeycomb, incident.io) are first-party incident write-ups.
The Slack 2016 incident is documented inside an engineering blog post rather than a
standalone postmortem, so it is graded
blogdespite carrying incident detail. - RabbitMQ PR #2129 is the closed-unmerged design argument this skill hunts for; PR #12906 is its counterpoint, a flow-control regression restored after a user report.
- Vendor material (NATS docs, SQS pricing) is used only for defaults and prices, and is labelled as vendor in the page.
- No public postmortem in this corpus attributes an outage to work lost from a bounded-and-dropping queue (the NATS pole). Absence noted in the guide.