A GPU platform must decide between holding warm capacity and accepting cold starts. How should the two be compared economically?

The two costs to compare

Warm capacity costs idle hardware continuously. For expensive accelerators this is a large, predictable, recurring cost proportional to how much is held and for how long.

Cold starts cost latency, which converts to lost requests, worse conversion, or a lower price the platform can charge. That cost is variable, harder to measure, and concentrated on a minority of requests — which is why it is systematically underestimated.

The comparison in practice

• Estimate the warm cost directly: instances held × hours × rate. Straightforward.
• Estimate the cold-start cost by measuring what proportion of requests hit a cold start, what latency they experience, and what that does to the outcome you care about — abandonment, retry, customer complaint, or a contractual latency commitment.
• Model the middle options, since the comparison is rarely binary: a small warm pool covering the popular subset, affinity routing, queue smoothing for batch-tolerant work, predictive prewarming ahead of known patterns.

The intervention that changes the arithmetic

Reducing the cold start itself, rather than choosing between the two. Snapshot and restore, lazy image pulls, node-local artefact caches and affinity routing all shrink the cold-start cost, which improves every option simultaneously.

This is engineering paid once against a warm-capacity cost paid continuously, and for expensive hardware the arithmetic favours the engineering almost immediately. Teams tend to invest in the scaling policy first because it is a configuration problem and in the cold start second because it is an engineering one.

The usage distribution that determines the answer

Model and function usage is heavily skewed: a handful account for most traffic. That makes a small warm pool disproportionately effective — covering the top few workloads warm and letting the tail cold-start captures most of the benefit for a fraction of the cost.

Designing as though usage were uniform produces either an unaffordable warm pool or an unnecessarily poor experience.

The product resolution

Expose the trade as priced tiers. A guaranteed-latency tier backed by reserved warm capacity, and a cheaper best-effort tier that may cold-start. That moves the decision to the customer, who knows whether their workload is latency-sensitive, and it converts an internal cost dilemma into a revenue segmentation.