Adding capacity under fire

Reconstructs, from postmortems at Slack, AWS, Datadog, Robinhood and Coinbase plus the Kubernetes project's own rejected pull requests, why the mechanism bought to handle demand spikes so often fails during them. Gives the four failure classes, the measured latency floor of the reactive loop, the decision conditions for pre-provisioning versus reacting versus shedding, and a seven-rung ladder for testing your own capacity path before a surge does.

What you get out of it

• Three of the four failure classes have nothing to do with reacting too slowly: the signal inverts under failure, the scale-up is itself a risky change, and the platform below the autoscaler is finite.
• The measured floor of fully reactive scaling on stock Kubernetes is roughly 5 to 8 minutes signal-to-serving; anything that ramps faster must be absorbed by pre-bought headroom or shed at admission.
• The most spike-exposed operators do not scale into their spikes: Shopify files capacity estimates with cloud providers months before BFCM and rehearses the scale-up; Coinbase survived 6x forecast error by throttling, not scaling.
• Recovery at scale is a load test of the cloud control plane you have never run: Datadog hit ENI rate limits (99% of calls), a 15,500-instance mesh limit and cluster maximums while replacing 60% of its fleet.
• Scaling velocity is business policy, not an implementation detail: the argument is recorded in kubernetes/kubernetes PRs #32408 and #34424 (rejected 2016) and reversed by KEP-853 (shipped v1.18, 2020).

Scope

Why this, now. The October 2025 us-east-1 event put a 12-hour hole under every autoscaler in the region, and the newest Kubernetes autoscaler is currently adding static-capacity and buffer features because users keep asking for ways to not autoscale.

What it does not cover. Scaling stateful stores, queue backlog management, retry-storm dynamics (covered by sibling digs), serverless cold starts, and GPU/LLM capacity economics.