Capacity Planning Under Lumpy Demand

CPU capacity planning works because demand aggregates: many small workloads sum to something close to smooth, and the law of large numbers does the rest. GPU demand does not behave this way. A single training run can be a large fraction of the fleet, it arrives as a step function, and it lasts for weeks. Planning for it is closer to project scheduling than to capacity forecasting.

The three demand classes

Serving is the most predictable: it follows traffic, which has daily and weekly seasonality and grows with product usage. It has a hard availability requirement and a latency target, so it needs headroom for peaks and for failure.

Scheduled training is periodic retraining, evaluation runs and pipelines. Predictable in shape, flexible in timing, and the natural candidate for filling troughs in serving demand.

Exploratory work is research, experimentation and one-off large runs. It is unpredictable in both timing and size, it is the class that produces the step functions, and it is the reason a fleet sized on average demand is always short.

The planning approach that follows is to size committed capacity to the sum of serving baseline and scheduled work, and to serve exploratory demand from a mix of burst capacity, interruptible capacity and a queue that makes waiting explicit rather than invisible.

The commitment mix

Reserved or committed capacity is substantially cheaper per hour and is paid for whether used or not. On-demand costs more and is paid only when used. The break-even utilisation is simply the ratio of the two prices: if committed capacity costs 60 percent of on-demand, it pays off above 60 percent utilisation.

The complication specific to GPUs is that availability, not price, is often the binding constraint. On-demand capacity for the current generation of accelerator may simply not exist in the region you want when you want it, so a commitment buys access rather than a discount. Teams that plan purely on price discover this at the worst moment.

Lead times reinforce it. Committed cloud capacity for scarce accelerator types is negotiated in quarters. On-premises hardware has procurement, datacentre and installation timelines measured in months. A capacity decision made today affects what is possible two quarters out, which means the forecast has to extend that far even though the exploratory demand class is not forecastable.

When it breaks

Averages hide the queue. A fleet at 70 percent average utilisation can have researchers waiting days for capacity, because demand arrives in large indivisible chunks. Queue wait time at the percentiles, rather than utilisation, is the metric that reflects the experience of the people using the cluster.

Utilisation targets create perverse incentives. Pushing toward high utilisation removes the slack needed to absorb failures and to start large jobs promptly. Above roughly 80 percent, queue times rise sharply for the same reason they do in any queueing system, and a fleet run at 95 percent allocation is one failed node away from an incident.

Generational transitions strand plans. A new accelerator generation changes price-performance enough that a commitment made on the old generation becomes expensive relative to what is available. Commitment terms and generation cadence interact, and shorter commitments cost more per hour precisely because they preserve that option.

The unit of capacity is not the GPU. Interconnect topology, memory per device, and host CPU and network provisioning all constrain what a fleet can actually run. A count of accelerators without their configuration does not determine which jobs fit, and plans expressed in GPU counts alone routinely acquire hardware that cannot host the intended workload.