Evidence ledger
28 sources
Checked 17 Sep 2026
Evidence ledger
One row per claim in Adding capacity under fire: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production systems get capacity in place when demand surges, and why reactive autoscaling keeps failing at exactly that moment.
Research date: 2026-09-17. All URLs were verified during this session. Verification method note: this run executed in a Claude Code web sandbox whose egress policy allows full fetches of github.com, raw.githubusercontent.com, datadoghq.com and googleapis.com only. Sources on those hosts were fetched in full and quotes are copied verbatim ("fetched"). Sources on other hosts were verified through repeated targeted web searches whose result snippets returned the quoted text ("search-verified"); the URL, title, publisher and date were confirmed through at least two independent search results, but the full page could not be fetched from inside the sandbox. Quotes from search-verified sources are reproduced as returned by search and may differ trivially in punctuation from the page.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Slack | Slack's Outage on January 4th 2021 | postmortem | 2021-02 | 2026-09-17 (search-verified) | https://slack.engineering/slacks-outage-on-january-4th-2021/ | During the outage, network degradation made web-tier CPU drop, and Slack's autoscaling scaled the web tier down in the middle of the incident | "Network problems meant that Slack's web servers were waiting for results from the backend, which caused CPU utilisation on those web servers to drop. This triggered some automated downscaling and web servers were shut down." (as summarised across search results quoting the post) |
| 2 | Slack | Slack's Outage on January 4th 2021 | postmortem | 2021-02 | 2026-09-17 (search-verified) | https://slack.engineering/slacks-outage-on-january-4th-2021/ | A concurrent thread-count signal triggered a huge scale-up that the provisioning path could not absorb | "Slack attempted to add 1,200 servers to its web tier between 7:01am PST and 7:15am PST"; provision-service "had created a large number of instances, but most of them were not fully provisioned and were not serving" |
| 3 | Slack | Slack's Outage on January 4th 2021 | postmortem | 2021-02 | 2026-09-17 (search-verified) | https://slack.engineering/slacks-outage-on-january-4th-2021/ | The managed network layer (AWS Transit Gateway) was supposed to scale transparently and did not; AWS scaled it manually | "Slack's Transit Gateways (TGWs) did not scale fast enough"; AWS engineers "increased TGW capacity manually"; the trigger was the annual cold-cache surge on the first Monday of the year |
| 4 | Lorin Hochstein (independent) | Slack's Jan 2021 outage: a tale of saturation | blog | 2021-02-08 | 2026-09-17 (search-verified) | https://surfingcomplexity.blog/2021/02/08/slacks-jan-2021-outage-a-tale-of-saturation/ | Independent corroborating analysis of the Slack incident as a chain of saturations | Post analyses the same incident as saturation cascading across TGW, web tier and provision-service |
| 5 | AWS | Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region (Nov 2020) | postmortem | 2020-11 | 2026-09-17 (search-verified) | https://aws.amazon.com/message/11201/ | Adding capacity was itself the trigger: new front-end servers pushed the all-to-all connection mesh past the OS thread limit | The capacity addition "caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration"; capacity was added 2:44–3:47 AM PST; the event ran roughly 17 hours |
| 6 | AWS | Summary of the Amazon DynamoDB Service Disruption in US-EAST-1 (Oct 19–20, 2025) | postmortem | 2025-10 | 2026-09-17 (search-verified) | https://aws.amazon.com/message/101925/ | After DynamoDB recovered, EC2's DropletWorkflow Manager entered congestive collapse and new instance launches in us-east-1 were impaired for roughly 12 more hours, so anything that planned to scale or replace its way out of the incident could not | DWFM lease re-establishment across "hundreds of thousands" of droplets "could not complete before leases timed out, leading to a state of congestive collapse"; recovery required throttling incoming work and selectively restarting DWFM hosts; NLB health checks flapped on newly launched instances |
| 7 | Datadog | 2023-03-08 incident: A deep dive into the platform-level impact | postmortem | 2023-05-24 | 2026-09-17 (fetched) | https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-impact/ | A routine systemd update disconnected the majority of the fleet, and the cloud's health automation amplified it by mass-replacing instances | "By 08:00, AWS had terminated and replaced approximately 60 percent of the instances in the region, representing tens of thousands of instances." |
| 8 | Datadog | 2023-03-08 incident: A deep dive into the platform-level recovery | postmortem | 2023-06 | 2026-09-17 (fetched) | https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-recovery/ | Recovery-at-scale ran into the platform's own limits: VPC mesh instance limit, API rate limits, and cluster maximums | "We left off with the realization that we had lost 60 percent of our compute capacity."; "at 14:18 UTC we had reached the limit of 15,500 instances for our mesh"; "more than 99 percent of the calls were rate limited between 06:00 and 15:00 UTC" (ec2:CreateNetworkInterface); "The cluster-autoscaler kept adding nodes until we reached the maximum capacity for the three clusters (6,000 nodes per cluster)." |
| 9 | Datadog | 2023-03-08 incident: A deep dive into the platform-level recovery | postmortem | 2023-06 | 2026-09-17 (fetched) | https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-platform-level-recovery/ | The autoscaler had to be turned off mid-incident to protect a saturated dependency (Vault), then re-enabled progressively | Cluster-autoscaler disabled at 16:31 UTC to reduce load on Vault; re-enabled 18:34–19:00 UTC; nodes "take longer than usual to register because control planes are unstable and (mostly) because Vault is being rate limited" |
| 10 | Datadog | 2023-03-08 Incident: Infrastructure connectivity issue affecting multiple regions | postmortem | 2023-03 | 2026-09-17 (fetched) | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | Mass node replacement is itself a thundering herd against the cloud provider | "thousands of nodes being replaced at once, it also created a thundering herd that tested the cloud provider's regional rate limits in various ways, none of which was obvious ex ante" |
| 11 | Robinhood | An Update From Robinhood's Founders | postmortem | 2020-03-03 | 2026-09-17 (search-verified) | https://robinhood.com/us/en/newsroom/an-update-from-robinhoods-founders/ | Record demand produced infrastructure stress that cascaded into DNS failure; capacity did not exist to absorb it | "stress on our infrastructure—which struggled with unprecedented load. That in turn led to a 'thundering herd' effect—triggering a failure of our DNS system." |
| 12 | Netflix | Scryer: Netflix's Predictive Auto Scaling Engine | blog | 2013-11-05 | 2026-09-17 (search-verified) | https://netflixtechblog.com/scryer-netflixs-predictive-auto-scaling-engine-a3f8fc922270 | Netflix documented in 2013 that reactive scaling cannot beat provisioning lag, and that outage-driven traffic drops trigger scale-down right before the retry storm | Instance startup times "range from 10 to 45 minutes"; a traffic drop from an outage "usually triggers a down scale event, leaving the system under provisioned to handle the ensuing retry storm" (per the post's list of reactive-scaling weaknesses) |
| 13 | Netflix | Netflix/aws-autoscaling (tools and docs) | source | 2013 (repo) | 2026-09-17 (fetched) | https://github.com/Netflix/aws-autoscaling | Netflix published its autoscaling tooling and lessons as a public repo alongside Scryer | Repo title: "Tools and Documentation about using Auto Scaling" |
| 14 | Shopify | Capacity Planning at Scale | blog | ~2021 (Kir Shatrov, Production Engineering) | 2026-09-17 (search-verified) | https://shopify.engineering/capacity-planning-shopify | For the largest known spike of the year, Shopify plans capacity ahead with its cloud providers and rehearses scale-ups rather than trusting reactive scaling | Shopify "submits these estimates to their cloud providers early so the providers can ensure they have enough physical infrastructure available"; rehearsal scale-ups named Mayday (2019), Spooky scale-up (2019), Oktoberfest scale-up (2020) |
| 15 | Shopify | How we prepare Shopify for BFCM (2025) | blog | 2025-11 | 2026-09-17 (search-verified) | https://shopify.engineering/bfcm-readiness-2025 | Scale of the rehearsed event, and load testing on production from three regions | BFCM 2024 peaks reported: 284M requests/min at edge, 80M requests/min on app servers, 12 TB/min egress; load-testing tool Genghis runs scripted buyer workflows against production from three GCP regions |
| 16 | Coinbase | Everyone Wins at Super Bowl LVI — WAGMI! | postmortem | 2022-02 | 2026-09-17 (search-verified) | https://blog.coinbase.com/everyone-wins-at-super-bowl-lvi-wagmi-c0039452975f | A 60-second TV ad produced a spike 6x beyond load-tested benchmarks in under a minute; no autoscaler acts on that timescale, and the mitigation was throttling | "over 20M+ hits on their landing page in one minute"; "engagement that was 6 times higher than our previous benchmarks"; the team "temporarily throttled" traffic to stabilise |
| 17 | Autopilot: workload autoscaling at Google (EuroSys '20) | paper | 2020-04 | 2026-09-17 (search-verified) | https://dl.acm.org/doi/10.1145/3342195.3387524 | Even at Google, autoscaling's win is measured in reclaimed slack, not magic elasticity: autoscaled jobs run at 23% slack versus 46% for manual limits | "Autopiloted jobs have a slack of just 23%, compared with 46% for manually-managed jobs"; Autopilot "reduces the number of jobs severely impacted by OOMs by a factor of 10" | |
| 18 | Meta (Facebook) | Kraken: Leveraging Live Traffic Tests to Identify and Resolve Resource Utilization Bottlenecks (OSDI '16) | paper | 2016-11 | 2026-09-17 (search-verified) | https://www.usenix.org/conference/osdi16/technical-sessions/presentation/veeraraghavan | The only way Facebook trusted its capacity numbers was continuously load-testing production with live user traffic | Kraken "runs load tests by continually shifting live user traffic to one or more data centers"; in production three years; "allowed us to improve our hardware utilization by over 20%" |
| 19 | Kubernetes SIG Autoscaling | KEP-853: Configurable scale up/down velocity for HPA | adr | 2019 (implemented v1.18, 2020) | 2026-09-17 (fetched) | https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/853-configurable-hpa-scale-velocity/README.md | The project eventually accepted that scaling velocity is per-application policy; user story 1 is "Scale Up As Fast As Possible", and the KEP names the retry-spike problem stabilization addresses | "As a result, users cannot influence scale velocity, and that is a problem for many applications."; motivating app class: "should scale up as fast as possible (false positive signals to scale up are ok), and scale down very slowly" |
| 20 | Kubernetes | PR #32408: HPA: Allow customizing scale-forbidden-windows (closed unmerged) | source | 2016-09-09 | 2026-09-17 (fetched) | https://github.com/kubernetes/kubernetes/pull/32408 | The first attempt (by an HPA maintainer) to make scaling velocity configurable was rejected on API-purity grounds | Reviewer: "this PR exposes internal details of HPA algorithm in the API. We cannot expose them, as the algorithm may be changed in the future... We don't want to engrave internal stuff." |
| 21 | Kubernetes | PR #34424: move scale forbidden windows to hpa spec (closed unmerged) | source | 2016-10-10 | 2026-09-17 (fetched) | https://github.com/kubernetes/kubernetes/pull/34424 | Second attempt, same rejection, with the argument recorded | "This idea already appeared in #32408: scale-up/scale-down windows are internal details of the autoscaling algorithm, so, we don't want to expose it in the API." |
| 22 | Kubernetes | Issue #56335: programmable waiting time before autoscale-up/down | source | 2017-11-24 | 2026-09-17 (fetched) | https://github.com/kubernetes/kubernetes/issues/56335 | The design argument continued for years; maintainers held the line, then conceded the underlying problem | "the conclusion has always been that the forbidden window is an implementation detail that we really shouldn't need at all, so we shouldn't expose it as a knob" (DirectXMan12); later: "I'm not convinced we want to expose the fields as requested in this issue, but the problem of arbitrary forbidden periods and 2x limit on scale-up is still there and needs to be addressed." |
| 23 | Kubernetes | Issue #39090: HPA should have scale down/up limits | source | 2016-12-21 | 2026-09-17 (fetched) | https://github.com/kubernetes/kubernetes/issues/39090 | Users asked for per-application scale-rate limits from 2016; KEP-853 cites this issue as motivation | Feature request thread referencing the forbidden-window discussion; cited in KEP-853's motivation alongside #65097 and #69428 |
| 24 | Kubernetes SIG Autoscaling | Cluster Autoscaler FAQ | source | living doc | 2026-09-17 (fetched) | https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md | The measured floor of reactive scaling in Kubernetes, the quota backoff, and the deliberate halt when the cluster looks broken | "No more than 30 sec latency on small clusters... No more than 60 sec latency on big clusters (100 to 1000 nodes)"; HPA "can be up to 3 minutes before pods are added or deleted"; "on GCE it usually takes 3 to 4 minutes from CA request to when pods can be scheduled on newly created nodes"; on quota failure CA "may wait up to 30 minutes before next attempt"; with >45% unready nodes "CA stops all operations until the situation improves" |
| 25 | Kubernetes SIG Autoscaling / Airbnb | Cluster Autoscaler proposal: Expander Plugin over gRPC | adr | ~2021 | 2026-09-17 (fetched) | https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/proposals/expander-plugin-grpc.md | Airbnb needed scaling decisions to encode business logic and proposed a pluggable decision point rather than forking | "we (Airbnb) would like to iterate on custom expander strategies (such as a weighted random expander, custom spot market strategy, etc) out of band with CA" (author: Evan Sheng, Airbnb) |
| 26 | Airbnb | Dynamic Kubernetes cluster scaling at Airbnb | blog | 2022-05 | 2026-09-17 (search-verified) | https://medium.com/airbnb-engineering/dynamic-kubernetes-cluster-scaling-at-airbnb-d79ae3afa132 | Production users outgrow the autoscaler's built-in decision strategies and plug in their own | Default expanders "were not sophisticated enough for their complex business requirements around cost and instance type selection"; built the gRPC custom expander of source #25 |
| 27 | Kubernetes SIG Autoscaling (Karpenter) | Karpenter design: Static Capacity | adr | checked from main branch | 2026-09-17 (fetched) | https://github.com/kubernetes-sigs/karpenter/blob/main/designs/static-capacity.md | The newest autoscaler is adding explicitly non-elastic capacity because users demand it | "several important use cases require maintaining a fixed set of nodes such as: 1. Performance-critical applications where just-in-time provisioning latency is unacceptable" |
| 28 | Kubernetes SIG Autoscaling (Karpenter) | Karpenter RFC: Capacity Buffer Support | adr | checked from main branch | 2026-09-17 (fetched) | https://github.com/kubernetes-sigs/karpenter/blob/main/designs/capacity-buffers.md | Just-in-time provisioning latency drove a first-class headroom feature; the community asked at least five times | "Karpenter provisions nodes just-in-time based on pending pod demand. This creates scheduling latency as pods wait for node provisioning. Users need the ability to pre-provision spare capacity"; cites issues #749, #987, #3240, #3384, #4409 |
| 29 | AWS | Amazon EC2 Auto Scaling introduces Warm Pools | vendor | 2021-04-08 | 2026-09-17 (search-verified) | https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ec2-auto-scaling-introduces-warm-pools-accelerate-scale-out-while-saving-money | The cloud's own answer to launch latency is pre-initialised stopped instances; the latency claim is the vendor's | Pre-initialised stopped instances "can start to serve traffic in as low as 30 seconds"; aimed at apps with initialisation "that can take several minutes or more" |
| 30 | AWS | EC2 Auto Scaling User Guide: Warm pools (docs mirror) | vendor | living doc | 2026-09-17 (fetched, GitHub mirror) | https://github.com/awsdocs/amazon-ec2-auto-scaling-user-guide/blob/master/doc_source/ec2-auto-scaling-warm-pools.md | Operational mechanics of warm pools (lifecycle hooks, pool states) | Docs mirror fetched from the awsdocs GitHub organisation |
| 31 | Datadog | Datadog Container Report (Nov 2023) | casestudy | 2023-11 (data from Sept 2023) | 2026-09-17 (fetched) | https://www.datadoghq.com/container-report/ | Measured fleet-wide: autoscaling is widely adopted, and most workloads still hold double the capacity they use | "Over half of Kubernetes organizations have adopted Horizontal Pod Autoscaling"; "over 65 percent of Kubernetes workloads are utilizing less than half of their requested CPU and memory"; "less than 1 percent of Kubernetes organizations use VPA" |
| 32 | CAST AI | 2024 Kubernetes Cost Benchmark Report | casestudy | 2024 | 2026-09-17 (search-verified) | https://cast.ai/press-release/cast-ai-analysis-finds-only-13-percent-of-provisioned-cpus-and-20-percent-of-memory-is-utilized/ | Independent corroboration of massive over-provisioning in autoscaled fleets | For clusters of 50+ CPUs, "only 13 percent of provisioned CPUs and 20 percent of memory is utilized" (vendor-adjacent analysis of customer clusters) |
| 33 | Datadog / USENIX | The World Blew Up but We're All Okay (SREcon23 EMEA) | talk | 2023-10 | 2026-09-17 (search-verified) | https://www.usenix.org/conference/srecon23emea/presentation/de-vesine | Conference account of the March 2023 recovery: coordination scale of recovering 60% of a fleet | Laura de Vesine & Laurent Bernaille; "successfully coordinating more than 500 engineers over 2+ days of continual response"; slides at usenix.org (srecon23emea-slides_devesine.pdf) |
| 34 | The Downtime Project | Ep. 1: Slack vs TGWs | talk | 2021 | 2026-09-17 (search-verified) | https://downtimeproject.com/podcast/episode-1-slack-vs-tgws/ | Independent practitioner walkthrough of the Slack incident, including the scale-up-that-didn't-serve failure | Episode dedicated to the Jan 4 2021 outage; companion episode "Kinesis Hits the Thread Limit" covers source #5 |
Talk citation caveat
The two talk-tier sources (#33, #34) are cited to their programme pages and episode pages. The skill prefers timestamp-level citation into the recording; the sandbox's egress policy made the recordings unfetchable, so claims taken from them are limited to what their abstracts and slides metadata state. This is disclosed in the guide's evidence section.