The Cost Floor of Inference
What sets the minimum achievable cost per token, why memory bandwidth rather than compute is the binding constraint, and which techniques move the floor rather than approaching it.
There is a physical minimum cost to producing a token with a given model on given hardware, and most deployments operate well above it. Knowing where the floor is tells you how much of a cost problem is engineering and how much is architectural.
What sets the floor
Decode is memory-bandwidth-bound. Each token requires reading the model weights and the relevant KV cache from memory, so the time per token is bytes moved divided by achieved bandwidth, and the cost per token is that time multiplied by the hourly cost of the device.
Two things immediately follow.
Batching is the dominant lever. A weight read at batch size one serves one sequence; the same read at batch size 64 serves 64. Cost per token therefore falls close to linearly with batch size until either memory for the KV cache runs out or the operation becomes compute-bound. This is why serving economics are dominated by concurrency, and why a system that cannot batch effectively pays many times the floor.
The KV cache bounds the batch. Cache size per sequence grows with context length, so at long context the number of concurrent sequences that fit is set by cache memory, not by weights. Long-context serving is expensive primarily for this reason rather than because attention costs more compute.
Moving the floor versus approaching it
Approaching the floor is an engineering problem: continuous batching so slots are never idle, paged cache management so memory is not fragmented, prefix caching so shared prompts are not recomputed, disaggregating prefill from decode so each runs at its own optimal batch size, and scheduling that keeps utilisation high. A well-engineered stack can be several times cheaper than a naive one on identical hardware and an identical model.
Moving the floor requires changing what must be read. Quantising weights reduces the dominant term directly. Grouped-query attention and latent attention reduce cache bytes. Sparse activation through mixture of experts reduces the weights read per token, though all experts must remain resident. Smaller models move it furthest and change what the system can do.
The distinction matters because the two are pursued by different people at different times, and a team asking for a cheaper model when their stack is three times above the floor is solving the wrong problem.
When it breaks
Latency and cost pull apart. Larger batches lower cost per token and raise per-request latency, since a request waits for its batch. The operating point is a product decision, and a system tuned for minimum cost will not meet an interactive latency target.
Variable load wastes the batch. Achieving high average batch size requires steady demand, so a workload that is bursty runs at low batch during troughs and pays accordingly. Queuing, mixing batch and interactive workloads, and autoscaling all address this and none makes it free.
The floor assumes the hardware is well matched. Serving a small model on a large accelerator wastes memory and bandwidth capacity that the model cannot use, so the achievable cost per token is worse than the same model on appropriately sized hardware.
Prefill and decode have different floors. Prefill is compute-bound and its cost scales with prompt length; decode is memory-bound and scales with output length. A single cost-per-token figure blends two different economics, and workloads with long prompts and short outputs sit almost entirely in the first.
12 flashcards for this concept
Click a card to reveal the answer.