The mechanism sold as the answer to demand spikes keeps failing during them, in ways that Netflix wrote down in 2013 and that Slack, AWS, Datadog, Robinhood and Coinbase have re-demonstrated since. This guide reconstructs how spike-surviving systems actually get capacity in place, and gives you the conditions under which reactive scaling is the wrong tool for the very event it is named after.
The problem, stated without naming a product: when demand rises faster than the capacity behind a service, something must add capacity before queues and timeouts turn the surge into an outage. The record of who wins that race, and how, is unusually public.
Start with the surprise, because it reframes everything below. In November 2013, Netflix published the Scryer post, listing the failure modes of reactive autoscaling: instances take 10 to 45 minutes to come up, so capacity arrives after the spike; and an outage makes traffic drop, which "triggers a down scale event, leaving the system under provisioned to handle the ensuing retry storm". Eight years later, on 4 January 2021, Slack's postmortem describes exactly that script running in production: network degradation made web-server CPU drop, automated downscaling shut healthy web servers off in the middle of the incident, and the panicked scale-up that followed buried the provisioning service. The failure mode was published, named and mitigated at one company before most of the industry had adopted the tool that exhibits it.
The second thing the record shows: the teams facing the largest, most predictable spikes do not scale into them. Shopify submits capacity estimates to its cloud providers months ahead and rehearses the scale-up under names like Mayday and Oktoberfest; Netflix built prediction because reaction was too slow; and the newest-generation Kubernetes autoscaler, Karpenter, is adding static capacity and capacity buffers because its users keep asking for ways to not autoscale. Reactive scaling is the tool for the gentle slope: diurnal curves, gradual growth, cost recovery after the peak. For the cliff, production systems use headroom bought in advance, prediction, and load shedding, in that order.
Scope: this guide covers compute capacity under demand surge, meaning the loop from load signal to serving instance, at the VM and container level. It deliberately does not cover scaling stateful stores (a separate dig in this collection covers live resharding), queue backlog management, retry-storm dynamics (also covered separately), serverless cold starts, or GPU/LLM capacity, which has its own economics.
No single company runs the whole reference shape below, but every component appears in at least two published systems, and the divergences are themselves decisions (section 03).
The signal plane is slower than most architects assume. In stock Kubernetes, kubelet scrapes CPU every 10 seconds, Metrics Server collects every minute, the HPA evaluates every 30 seconds and then backs off; the Cluster Autoscaler FAQ puts the honest total at "up to 3 minutes before pods are added or deleted, but usually it's closer to 1 minute". That is the input delay before any decision exists. The systems that survive spikes add a second input that has no delay at all: a calendar. Shopify's capacity planning starts from forecast models and merchant growth projections and, critically, is communicated to the cloud providers early so the physical machines exist in the right regions. Netflix's Scryer generated a per-minute plan from traffic history and drove scaling ahead of need. Prediction here is not exotic machine learning; the highest-value forecast in the record is "Black Friday is on November 28" and "US markets open at 9:30".
The decision plane is where the industry spent a decade arguing, and section 03 covers the argument. The load-bearing point: velocity and stabilization policy (how fast up, how fast down, how long to distrust a dip) is a separate concern from the target-tracking math, and it is per-application, because a checkout tier and a batch fleet have opposite risk profiles. Kubernetes only conceded this in v1.18 (2020) with the behavior field, whose first user story is titled "Scale Up As Fast As Possible" and explicitly accepts false positives on the way up.
The actuation plane is a supply chain, not a function call. A scale-out decision traverses the cloud control plane (which enforces quotas and API rate limits), instance provisioning (the Cluster Autoscaler's measured figure is 3 to 4 minutes on GCE from request to schedulable node), then your own bootstrap: config management, secret issuance, cache warming. Slack's January 2021 outage is the canonical demonstration that this last stage is a production dependency with its own capacity: 1,200 instances were created and "most of them were not fully provisioned and were not serving", because provision-service itself ran over the same degraded network it was trying to route around. Warm pools attack the middle of the chain: AWS claims pre-initialised stopped instances serve "in as low as 30 seconds" (a vendor figure, from the 2021 launch); Karpenter's capacity-buffers RFC generalises the same idea inside Kubernetes after the community requested it at least five times.
The protection path is the component that reframes the whole problem. Capacity cannot arrive in seconds; admission control can act in seconds. When Coinbase's Super Bowl QR code produced 20 million hits on the landing page in a minute, six times anything they had load-tested, what stabilised the site was throttling, not scaling. The architecture conclusion, stated plainly: shedding is the fast path and scaling is the slow path, and a design that has only the slow path has decided, implicitly, to fail closed for everyone instead of degrading for some. (This guide's sibling on queue backlogs covers the shedding mechanics.)
Finally, validation. Facebook's Kraken paper (OSDI 2016) describes continuously load-testing production by shifting live user traffic between data centers, because synthetic tests kept lying about real capacity; the paper credits the practice with over 20% hardware-utilization improvement across three years. Shopify runs the same loop yearly with its Genghis load generator and named scale-up rehearsals. Inferred, but hard to avoid: a scale-up path that has never been exercised at full size is untested code on the critical path, and the incidents in section 04 are what its first execution looks like.
Each block is a fork the sources actually argued about, with the condition that flips the answer. The middle one contains a rare artefact: a maintainer position held for three years, recorded in rejected pull requests, then reversed.
behavior field in v1.18| Decision | Common choice | Alternative | Because | Evidence |
|---|---|---|---|---|
| Scaling signal | CPU utilisation | Concurrency, queue depth, request rate | CPU is universal but inverts under dependency failure; Slack ran a CPU-based scale-down and a thread-count scale-up simultaneously during one incident | Slack, 2021 |
| Headroom target | Run hot, reclaim slack (Google Autopilot: 23% slack vs 46% manual) | Run cold into known peaks (Shopify BFCM) | Slack reclaimed is money; but headroom must exceed what can arrive inside your lead time | Autopilot, 2020 · Shopify |
| Launch latency | Warm pools / capacity buffers | Faster boot, bigger instances | Pre-initialised instances cut minutes of bootstrap to a claimed 30 seconds; buffers moved into the autoscaler itself after repeated user demand | AWS, 2021 · Karpenter RFC |
| Static floor | Pure elasticity, min-replicas only | Explicit static capacity in the autoscaler | Karpenter is adding static NodePools for "performance-critical applications where just-in-time provisioning latency is unacceptable" | Karpenter design |
| Scaling decision logic | Built-in strategies | Pluggable business logic | Airbnb outgrew the five built-in expanders and contributed a gRPC plug-point rather than forking | CA proposal · Airbnb, 2022 |
| Overflow handling | Queue and wait for capacity | Admission control and shed | Capacity arrives in minutes; throttling acts in seconds and kept Coinbase partially up at 6x tested load | Coinbase, 2022 |
Six published incidents, grouped by the four failure classes of Figure 1. Notice how rarely "the autoscaler reacted too slowly" is the root cause; the deeper pattern is that the capacity machinery itself became the failing system.
Latencies of the capacity loop, magnitudes of real spikes, and the cost of the alternative. Measured, claimed and derived are separated; undated numbers are not offered.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Metrics-to-HPA-action delay | up to 3 min | Kubernetes defaults | Scrape 10s, aggregate 1 min, 30s control loop; "usually closer to 1 minute" (measured behaviour of defaults) | 2026-09 | CA FAQ |
| Cluster Autoscaler decision latency | ≤30–60 s | SIG Autoscaling | Measured SLO: 30s small clusters (avg 5s), 60s at 100 to 1,000 nodes (avg 15s), no pod affinity | 2026-09 | CA FAQ |
| Node provisioning | 3–4 min | GCE, measured | From CA request to pods schedulable on the new node | 2026-09 | CA FAQ |
| EC2 instance startup | 10–45 min | Netflix, measured | The number that motivated predictive scaling; modern instances are faster, the 2013 figure is the historical bound | 2013-11 | Scryer |
| Warm-pool instance to serving | ~30 s | AWS | Vendor claim at launch, "as low as"; for apps with minutes-long initialisation | 2021-04 | AWS |
| CA backoff after failed scale-up (e.g. quota) | up to 30 min | Kubernetes | Documented backoff between retry attempts when the provider refuses capacity | 2026-09 | CA FAQ |
| CA fail-static threshold | 45% / 3 nodes | Kubernetes | Above this many unready nodes, "CA stops all operations until the situation improves" | 2026-09 | CA FAQ |
| Mid-incident scale-up attempt | 1,200 servers / 14 min | Slack | Instances created but mostly never serving; provisioning path saturated | 2021-01 | Slack |
| Ad-driven spike | 20M+ hits/min, 6x benchmarks | Coinbase | Landing page, 60-second Super Bowl ad; stabilised by throttling | 2022-02 | Coinbase |
| Rehearsed peak | 284M req/min edge, 80M app | Shopify | BFCM 2024 peaks, 12 TB/min egress; pre-provisioned and load-tested for months | 2025-11 | Shopify |
| Fleet lost to one change | ~60% | Datadog | "Tens of thousands of instances" terminated and replaced by AWS health automation within ~2 h | 2023-03 | Datadog |
| Recovery-path API throttling | >99% of calls | Datadog on AWS | ec2:CreateNetworkInterface rate-limited 06:00 to 15:00 UTC during mass replacement | 2023-03 | Datadog |
| Regional launch impairment | ~12 h | AWS us-east-1 | EC2 launches failing or degraded after DynamoDB DNS repair, DWFM congestive collapse | 2025-10 | AWS |
| Capacity add that triggered a 17 h outage | 63 min of adds | AWS Kinesis | Fleet-wide OS thread limit crossed; front-end restart was the long pole | 2020-11 | AWS |
| Autoscaled slack vs manual | 23% vs 46% | Autopilot-managed jobs vs manually-limited jobs; OOM-impacted jobs cut 10x | 2020-04 | Autopilot | |
| Workloads using <50% of requests | >65% | Datadog customer fleet | Measured across billions of containers; HPA adopted by over half of K8s orgs | 2023-11 | Container Report |
| Provisioned CPU actually used | 13% | CAST AI customers | Clusters of 50+ CPUs; memory 20%; the 2025 edition reports 10% CPU | 2024 | CAST AI |
| Utilisation gained by live-traffic testing | >20% | Kraken shifting live traffic to find real capacity ceilings, over three years | 2016-11 | Kraken |
The 30-second warm-pool figure is a vendor launch claim, not an independent measurement; no published third-party benchmark of it surfaced in this hunt. The 10-to-45-minute EC2 startup figure is real but from 2013; treat it as an upper bound and measure your own. The utilisation figures (13%, 65%) come from vendors whose products monitor or optimise clusters, so the sample skews toward teams that already suspected waste. Derived, not reported: summing the pipeline stages in Figure 3 gives roughly 5 to 8 minutes signal-to-serving for stock Kubernetes on GCE; nobody publishes their end-to-end number, which is why rung 1 of section 07 has you measure yours.
Every source behind this page, graded. One methodological note: this guide was researched from a sandbox whose network policy allowed full fetches of GitHub and datadoghq.com; sources on other hosts were verified through multiple independent search results rather than direct fetches, and the two talks are cited to their programme pages rather than to timestamps in the recordings. The ledger shipped beside this page (sources.md) marks the verification method per source.
The fullest public account of autoscaling behaving badly mid-incident: CPU-triggered scale-down of healthy servers, thread-triggered 1,200-instance scale-up, and a provisioning service sharing fate with the broken network.
A routine capacity addition crossed an invisible fleet-wide OS thread limit and took 17 hours to unwind. The scale-up itself was the change that broke production.
The DNS race is the headline; the capacity lesson is below it: EC2's droplet-lease manager entered congestive collapse and regional instance launches were impaired for roughly 12 hours after the trigger was fixed.
How a systemd security update deleted CNI routes on 60% of instances across three clouds, and how AWS health automation turned disconnection into mass termination.
The best published record of re-provisioning at scale: rate limits on 99% of ENI calls, a 15,500-instance mesh limit, cluster maximums, Vault saturation, and the decision to switch the autoscaler off.
Thin by design, and useful for exactly that reason: "unprecedented load", a "thundering herd", a DNS failure, and no architectural detail. The thinnest postmortem in this wall maps the limits of the public record.
The founding document of this topic. Names the provisioning lag (10 to 45 minutes) and the outage-then-retry-storm trap that reactive scaling walks into, then builds prediction from traffic history instead.
Kir Shatrov on how Shopify sizes for BFCM: forecast models, estimates filed with cloud providers early enough for physical hardware to exist, and rehearsal scale-ups with names (Mayday, Spooky, Oktoberfest).
The current edition of the yearly readiness account: 284M requests/min edge peak in 2024, production load testing from three regions with the Genghis tool, and months of failover rehearsal for a four-day event.
The company's own account of the QR-code ad: 20M+ hits in a minute, six times previous benchmarks, load tests beaten by reality, throttling as the stabiliser.
What outgrowing the autoscaler looks like: the built-in node-selection strategies could not express Airbnb's cost and instance-type logic, so they built the gRPC expander plug-point and upstreamed it.
Independent close reading of the Slack postmortem as a chain of saturations, valuable as a second analytical perspective not written by the affected company.
The measured case for autoscaling as a cost tool: autopiloted jobs run at 23% slack versus 46% for manual limits, with OOM-impacted jobs reduced 10x, across Google's fleet.
Facebook's answer to "what is our real capacity": continuously shift live user traffic to load individual systems and regions, with user-experience metrics as the guardrail; credited with over 20% utilisation improvement.
The decision record that ended the velocity argument: names the application classes (fast-up slow-down web, fast-both batch), cites the 2016 issues, and ships the per-workload behavior API in v1.18.
Airbnb's recorded rationale for a pluggable scaling-decision point: business logic in node selection changes faster than the autoscaler's release train, and forking is the only alternative.
The just-in-time autoscaler adding fixed-size NodePools, motivated by "performance-critical applications where just-in-time provisioning latency is unacceptable" and by users faking it with placeholder pods.
Headroom as a first-class API: pre-provisioned spare capacity that scales with the workload, replacing balloon-pod workarounds. Cites five separate community requests.
An HPA maintainer's own attempt to expose velocity control, rejected: "We cannot expose them, as the algorithm may be changed in the future... We don't want to engrave internal stuff."
Second community attempt within a month, closed with a pointer to the first: the windows are "internal details of the autoscaling algorithm". The demand signal was already unmistakable in 2016.
The thread where the position is stated most crisply ("we shouldn't expose it as a knob") and where it starts to crack ("the problem... is still there and needs to be addressed"). Issues #39090, #65097 and #69428 carry the same ask from 2016 onward.
The operational truth of the mainline autoscaler in one file: latency SLOs, the 3-to-4 minute GCE node figure, quota backoff up to 30 minutes, the 45%-unready halt, and the pause-pod overprovisioning recipe.
The tooling and documentation Netflix published alongside its autoscaling practice; period evidence that the lessons in Scryer were operationalised, not aspirational.
Fleet-scale measurement: over half of Kubernetes organisations run HPA, over 65% of workloads use less than half their requested CPU and memory, and under 1% use VPA.
Across customer clusters of 50+ CPUs: 13% of provisioned CPU and 20% of memory utilised. A vendor with an optimisation product to sell, so read as directional, but consistent with Datadog's independent measurement.
The platform's own admission that launch latency breaks elasticity for slow-booting apps: pre-initialised stopped instances, claimed to serve "in as low as 30 seconds". Mechanics in the user guide.
Laura de Vesine and Laurent Bernaille on running the March 2023 recovery: coordinating more than 500 engineers over 2+ days. Cited here to the programme page and slide deck; the recording was not fetchable from this research environment.
Practitioner podcast walkthrough of the Slack incident by engineers unaffiliated with Slack; the companion episode "Kinesis Hits the Thread Limit" covers the AWS event. Cited to the episode pages.
Seven rungs from a toy control loop to the drills the surge survivors run. The crossing from toy to real is rung 4, where you break your own signal.
A load generator, a pool of worker processes behind a queue, and a controller that adds a worker when average utilisation exceeds 70%. Add artificial provisioning delay (30 to 120 seconds) to mimic real instance startup.
Done when: you can chart demand, capacity and the gap over time, and state your end-to-end signal-to-serving lead time as one number. Teaches: the lead time is a property of the whole pipeline, not the controller.
Step the load 6x in ten seconds (the Coinbase shape). Compute the "capacity debt": the integral of unserved demand until the loop catches up.
Done when: you can predict the debt from the step size and lead time before running the test. Teaches: reactive scaling's loss is proportional to lead time; halving it halves the outage.
Implement KEP-853 semantics on your controller: separate up and down rules, a scale-down stabilization window, velocity caps. Re-run rung 2 plus a flapping load.
Done when: the checkout-profile config (instant up, slow down) and the batch-profile config (fast both ways) demonstrably differ in cost and in dropped requests. Teaches: velocity is per-workload business policy.
Make the workers' downstream dependency hang so utilisation collapses while queues grow (the Slack scenario). Watch your controller scale down into the failure, then fix it: gate scale-down on a health check independent of the load metric.
Done when: the dependency hang no longer shrinks the fleet, and a genuine load drop still does. Teaches: distinguishing loss-of-signal from loss-of-load is the core safety property.
Keep N pre-initialised workers stopped; measure spike response with and without. Compute the monthly cost of the pool against the capacity debt it eliminates.
Done when: you can state the break-even spike frequency at which the pool pays for itself. Teaches: headroom is a priced insurance product, not waste or virtue.
Inject quota errors and API rate limits into the provisioning stub (fail 99% of create calls, the Datadog number). Verify backoff behaviour, alerting, and what your controller does after 30 minutes of refusal.
Done when: a capacity-refused condition pages a human with the specific limit hit, instead of retrying silently. Teaches: the cloud is a dependency with its own capacity and its own bad days.
In production (or the closest staging you truly load), shift real traffic Kraken-style toward one instance group until a guardrail metric degrades. Record the true ceiling and re-run quarterly; rehearse the scale-up itself the way Shopify rehearses BFCM.
Done when: your capacity number comes from a drill log, not a spreadsheet, and the scale-up runbook has been executed while healthy. Teaches: untested capacity paths fail on first use, which is always during the surge.
The queries that found this material, grouped by what they surface. The highest-yield trick on this topic: search for the incident vocabulary ("scaled down", "thundering herd", "rate limited") next to the tool vocabulary, never the tool alone.
<company> outage postmortem autoscaling "scaled down""provision-service" slack outage "scale down" web tier"exceed the maximum number of threads" kinesispostmortem "cluster-autoscaler" "rate limited" OR "insufficient capacity""congestive collapse" site:aws.amazon.com OR EC2 launches postmortemrepo:kubernetes/kubernetes is:pr is:closed is:unmerged "forbidden window""KEP" site:github.com sig-autoscaling "scale velocity"karpenter designs static-capacity OR capacity-buffers site:github.comcluster-autoscaler proposals expander site:github.com"container report" utilization "less than half" requestedkubernetes cost report "provisioned CPUs" percent utilizedautopilot google eurosys "slack" workload autoscalingsite:shopify.engineering capacity planning OR BFCM readinessnetflix scryer predictive auto scalingkraken "live traffic" load testing osdi facebook"warm pools" EC2 auto scaling announcement