A serverless GPU platform has workloads with very expensive startup costs. How should predictive scaling, prewarming, scheduled capacity, autoscaling and queue-based smoothing be compared?
Show the full answer Hide the answer
Why cold start dominates the design
When starting a workload means pulling a large container image and loading a multi-gigabyte model into GPU memory, startup is measured in tens of seconds. That makes the scaling decision qualitatively different from web-tier scaling, where a new instance is useful in a second and the cost of being wrong is small.
Here, being wrong in either direction is expensive: an unnecessary warm instance is an idle GPU, and a missing one is a request waiting half a minute.
The options compared
| Strategy | Best when | Cost | Failure mode |
|---|---|---|---|
| Keep-warm pool | Latency-critical, predictable floor | Idle GPUs, continuously | Pool too small at peak |
| Predictive prewarming | Demand has a learnable pattern | Model complexity, wasted warms | Prediction misses an unusual event |
| Scheduled capacity | Known events | Simple, wasteful off-peak | Unscheduled spikes |
| Reactive autoscaling | Sustained ramps | Cheapest at steady state | Useless for fast bursts |
| Queue-based smoothing | Latency is negotiable | Almost free | Unacceptable if requests are interactive |
The techniques that attack the cold start itself
Rather than choosing between them, the highest-leverage work is making the cold start smaller:
- Snapshot and restore memory state instead of re-initialising, which is the difference between loading a model and resuming a process.
- Lazy and streaming image pulls, so execution begins before the whole image is present.
- Cache model weights on local storage on nodes likely to run that model, so the load is from disk rather than from object storage.
- Route by affinity — send a request for model X to a node that already has X resident — which converts most cold starts into warm ones without holding anything idle.
The judgement
Reducing cold start is worth more than optimising the scaling policy, because it makes every strategy better simultaneously and it removes the sharpest part of the trade-off. Teams tend to invest in the scaling policy first because it is a configuration problem, and in the cold start second because it is an engineering problem.
The final architecture is usually a small always-warm pool for the popular models, affinity routing, queue smoothing for batch-tolerant work, and predictive prewarming ahead of known patterns — with the fully reactive path as the backstop rather than the mechanism.