A live-streaming platform expects viewer count to increase tenfold within minutes when a major event begins. Its services autoscale on CPU. Predict what happens and what the design should be instead.

What happens with CPU-based autoscaling

It arrives too late and then overshoots.

The timeline is unforgiving. Viewers arrive over roughly sixty seconds. The metric pipeline reports CPU with a delay of tens of seconds. The scaler evaluates on an interval and applies a stabilisation window to avoid flapping. New instances must be provisioned, started, pass health checks, warm their caches and connection pools, and be added to the load balancer. Total time to useful capacity: several minutes.

During those minutes the existing fleet absorbs ten times its designed load. Latency rises, connections queue, some requests time out, clients retry — adding more load. By the time new capacity arrives, the system may already be in a self-sustaining overload that extra capacity does not resolve, because a large share of the traffic is now retries.

Then the event stabilises, CPU falls, and the scaler removes capacity — right before the next spike.

Why CPU is the wrong signal here

For a connection-heavy workload, CPU is a lagging and misleading indicator. A server holding many idle WebSocket connections has low CPU and is nonetheless near its memory and file-descriptor limits. The resource that binds is connections, not compute, and scaling on CPU means scaling on the wrong axis.

What the design should be

1. Pre-scale on the event calendar. Scheduled events are known. Capacity is provisioned before the event starts, based on a forecast, not on a metric. This is the single most important change, and it is organisational as much as technical: the platform needs to know what is scheduled.

2. Scale on leading indicators. Concurrent connections, queue depth and request arrival rate move before CPU does. Better still, scale on stream-start events and viewer-join rate — signals that predict load rather than reflect it.

3. Warm pools. Instances pre-provisioned and initialised, held ready. Adding them to rotation takes seconds rather than minutes. Costs idle capacity; buys the response time autoscaling cannot provide.

4. Static stability for the surge. The system should absorb the spike with capacity that already exists, treating autoscaling as a correction rather than as the mechanism. Anything that must succeed during the spike — a scaling API call, an image pull, a config fetch — is something that can fail during the spike.

5. Admission control as the backstop. When capacity genuinely runs out, shed deliberately: cap new viewer joins, degrade to lower bitrates, reduce chat features. A designed degradation beats an undesigned collapse.

6. Asymmetric scaling policy. Scale up fast and aggressively; scale down slowly and conservatively. The cost of excess capacity for an hour is trivial; the cost of removing capacity before a second spike is an outage.

The general principle

Autoscaling is a cost-optimisation mechanism, not a burst-survival mechanism. It handles the daily ramp and the gradual trend. Anything arriving faster than the provisioning loop must be met with capacity that is already there — which means forecasting, pre-scaling and a designed degradation path.