Cold-Start Amortisation

When starting a workload means pulling a large image and loading multi-gigabyte artefacts into memory, startup takes tens of seconds. That changes the scaling problem qualitatively: being wrong in either direction is expensive, because an unnecessary warm instance is idle expensive hardware and a missing one is a request waiting half a minute.

Teams respond by tuning the scaling policy. The higher-leverage work is making the cold start smaller, because that improves every strategy at once and removes the sharpest part of the trade-off.

Why it matters

Cold start is the constraint that determines whether serverless economics work at all for expensive workloads. A platform with a thirty-second cold start must hold capacity warm and therefore cannot be truly elastic; a platform with a two-second cold start can be.

Implementation patterns

• Snapshot and restore process memory rather than re-initialising. This is the difference between loading a model and resuming a process that had already loaded it, and it is usually the largest single improvement available.
• Lazy or streaming image pulls, so execution can begin before the whole image is present. Most of a large image is never read.
• Cache large artefacts on node-local storage for nodes likely to run that workload, so the load path is local disk rather than object storage across the network.
• Affinity routing: send a request for workload X to a node that already has X resident. This converts most cold starts into warm ones without holding anything idle, and it is the cheapest technique on the list.
• A small always-warm pool for the popular subset, since usage of models or functions is heavily skewed and a handful account for most traffic.
• Queue smoothing for batch-tolerant requests, keeping the warm pool for interactive ones.

Industry example

Inference platforms such as Replicate, Modal and Together AI all confront this: customers expect pay-per-use economics with interactive latency, on hardware whose initialisation is measured in tens of seconds. The techniques above are what makes the two compatible, and the ones that matter most are affinity routing and snapshotting — because they reduce the frequency and the cost of the cold start respectively.

The same problem appears outside AI wherever startup is expensive: JVM services with long warm-up, browser and device pools in testing platforms, and build sandboxes that must be isolated and therefore fresh.

Failure scenarios

• Optimising the scaling policy while ignoring the startup cost, which caps the achievable outcome.
• Warm pools sized for peak, which is simply pre-provisioning with extra steps and no elasticity.
• Affinity routing without a fallback, so a hot node becomes a bottleneck for a popular workload.
• Artefact caches with no eviction policy, filling node storage and causing failures unrelated to the workload.
• Cold-start latency invisible in metrics, because averages hide it — the affected requests are a minority and they are the ones customers complain about.

Trade-offs

Every technique adds complexity somewhere. Snapshotting constrains what a workload may do during initialisation and interacts badly with anything holding network connections or randomness. Affinity routing concentrates load and needs a spill-over path. Local caches consume node storage and need management.

The judgement is that these are engineering costs paid once, while the alternative — holding expensive capacity warm — is a cost paid continuously. For expensive hardware, the arithmetic favours the engineering almost immediately.

Interview question

"Your p50 inference latency is 400ms and your p99 is thirty seconds. Explain what is happening, and rank four things you would do about it by expected improvement per week of engineering effort."