Training Versus Inference Spend

Training costs attract the attention because they are large, discrete and quotable. Inference costs attract less and are usually the bigger number, because training happens once and inference happens on every request forever. Getting the relative magnitudes right determines which optimisations are worth engineering effort.

The crossover

Training cost is a one-off: parameters times tokens times the FLOP factor, divided by achieved throughput, times the hourly rate. Inference cost is per request and accumulates without limit.

The crossover point is where cumulative inference spend passes the training bill, and for a widely used model it arrives within months of launch. After that, every subsequent month is inference-dominated, and over a model's deployed life inference typically exceeds training by a large multiple.

The consequence is that a design choice trading training cost for inference cost is almost always worth making in the inference direction. Overtraining a small model, which spends more training compute to get a better model at a fixed inference cost, is exactly this trade and it is why it is standard practice. Distillation is the same trade. So is any architectural choice that reduces active parameters or KV cache size at the cost of a more complex training recipe.

Which optimisations pay

Ranked by lifetime impact for a high-volume deployment:

Reducing bytes per token moved at inference, through quantisation, KV cache compression and grouped-query attention, because decode is memory-bound and this is the dominant term.

Reducing tokens processed, through shorter prompts, prompt caching, retrieval that returns less, and avoiding unnecessary calls. Input tokens dominate in most retrieval and agent systems.

Routing to smaller models where they suffice, which is a large multiple on the requests it applies to.

Improving training efficiency, which matters for iteration speed and for the ability to run more experiments, and which affects a one-off cost.

Teams frequently invert this, spending heavily on training throughput because it is measurable and visible while leaving inference-side wins untaken.

When it breaks

Research organisations have the opposite profile. A lab running many training experiments and serving few requests is training-dominated, and the ranking above is wrong for them. The profile follows the deployment, not the technology.

Failed models never reach crossover. Training spend on models that are not deployed is pure cost, so the expected value calculation has to include the probability of shipping, which for exploratory work is well below one.

Fine-tuning changes the arithmetic. An organisation fine-tuning per customer has training costs that scale with customers rather than being one-off, which moves the crossover substantially and can invert the ranking.

Reasoning models shift the balance further toward inference. A model that emits many internal reasoning tokens per answer consumes far more inference compute per request than a single-pass model, which increases the inference share and raises the value of every token-reduction optimisation.