An AI inference platform's margin depends on GPU utilisation. Which levers actually change unit economics, and in what order?

Why utilisation is the business

When the dominant cost is expensive hardware billed by the hour, margin is almost entirely a function of how much of that hour does useful work. A platform at 40% utilisation has roughly half the margin of one at 80% on identical pricing, and no application optimisation matters as much.

The levers, in order of impact

1. Reduce cold starts. Every second spent loading a model is paid-for time producing nothing. Snapshot and restore, lazy image pulls, node-local artefact caches and affinity routing — sending a request for model X to a node already holding X — attack this directly, and affinity routing in particular converts most cold starts into warm ones without holding anything idle.
2. Dynamic batching. Combining concurrent requests into one forward pass is frequently a several-fold throughput improvement on the same hardware, at a small latency cost. The largest single lever available for inference, and the one that most directly converts latency tolerance into margin.
3. Right-size the hardware to the model. Running a small model on the largest accelerator is a pure margin loss, and the mapping from model to appropriate hardware is a catalogue decision rather than a scheduling one.
4. Bin-pack and reduce fragmentation. Capacity that exists but cannot be allocated because it is scattered in pieces too small is the dominant inefficiency in indivisible-resource scheduling, and consolidating small jobs is what accumulates contiguous free capacity.
5. Queue latency-tolerant work. A batch request that can wait smooths demand and fills troughs, which is free utilisation.
6. Tier the capacity: committed reserved capacity for the predictable baseline, on-demand for the middle, spot or preemptible for interruption-tolerant work with the price difference passed to the customer as a class.

What does not move the needle

Application-level optimisation of the serving code, which is a rounding error next to accelerator idle time. Micro-optimising the API layer. Reducing storage cost, which is negligible in this mix.

The pricing consequence

Per-request pricing on a resource with orders-of-magnitude cost variance subsidises the expensive minority. Pricing by compute time or by tokens aligns the incentive and — more usefully — gives customers a signal that changes their behaviour, which no technical control achieves as effectively.

The metric that should be on the engineering dashboard is cost per thousand inferences by model class, tracked over time. Total spend rises with growth and tells you nothing; unit cost falling while volume grows is the only evidence the efficiency work is working.