Autoscaling and Cold Starts in LLM Serving

The standard web autoscaling loop assumes a new replica is useful within seconds and that CPU utilisation tracks saturation. Neither holds for LLM serving. A new GPU replica needs to acquire an accelerator, pull tens of gigabytes of weights, load them into HBM, warm up kernels and capture CUDA graphs, and can take two to ten minutes before it serves its first token. Meanwhile the utilisation metric that would have triggered the scale-up reads 100 percent under normal healthy operation.

Autoscaling an LLM fleet is therefore a problem of predicting demand minutes ahead and of shortening a cold start that is dominated by data movement.

Why GPU utilisation is the wrong signal

Continuous batching exists precisely to keep the GPU busy. It pins utilisation near the top whether the server is comfortable or drowning, so the metric carries almost no information about headroom. The signals that do move before users notice are queue and memory pressure:

vLLM exports num_requests_waiting and gpu_cache_usage_perc for exactly this, and queue depth is the most common Kubernetes HPA custom metric in production LLM stacks. KV cache occupancy is the better early warning, because it saturates before the queue does.

Anatomy of a cold start

Roughly, in order of cost:

1. Capacity acquisition. Getting a GPU node from a cloud provider, when the instance type is available at all. Tens of seconds to minutes, and sometimes it simply fails.
2. Weight transfer. A 70B model in BF16 is about 140 GB. From object storage over a 10 Gbps link that is nearly two minutes at line rate. This is usually the dominant term.
3. Load and initialise. Host to device copy, tensor parallel sharding, allocator warm-up.
4. Warm-up. First-run kernel autotuning, CUDA graph capture, compile caches. Tens of seconds, and skipping it means the first requests are several times slower than steady state.

Mitigations attack the transfer term first: keep weights on local NVMe or a node-local cache rather than object storage, stream weights layer by layer so the model begins loading before the download finishes, keep a warm pool of idle replicas, and pre-pull images and weights onto nodes before they are needed.

Scale-to-zero and its price

Scaling a model to zero replicas is attractive for long-tail models: a fleet serving 200 fine-tunes cannot keep 200 replicas warm. The cost is that the first request after idle pays the full cold start, and no amount of engineering makes that acceptable for an interactive endpoint. Two ways out are common. Serve long-tail variants as LoRA adapters on a shared always-warm base model, which turns a cold start into an adapter load of a few hundred megabytes. Or accept scale-to-zero only for asynchronous and batch endpoints where a two-minute first-token latency is a queue delay rather than a failure.

When it breaks

• Scaling on a lagging signal. By the time p95 TTFT breaches the SLO, the replica you now request will arrive minutes after the spike has done its damage. Scale on queue depth and cache occupancy; alert on TTFT.
• Flapping. Aggressive scale-down followed by a cold start costs far more than the idle GPU it saved. Asymmetric policies are correct here: scale up fast, scale down slowly, with a generous stabilisation window.
• Draining is not instant. A replica marked for termination may hold sequences generating for another minute. Terminating it drops them unless the router stops sending new work and waits for the in-flight set to finish.
• Autoscaling cannot fix an underprovisioned floor. If the minimum replica count cannot absorb a normal spike within one cold-start window, the fleet is permanently reactive. The floor is a capacity decision, not an autoscaling one.