Architecting to a Latency Budget

"Make it smaller" is not a design brief. "Time to first token under 200 ms and 30 tokens per second per user at 64 concurrent requests on one A100" is, and the two lead to different models. Designing to a latency budget means starting from the memory system and working backwards, because on the operations that dominate inference the arithmetic is not the constraint.

The two regimes

Prefill processes the whole prompt at once. Every weight is loaded and multiplied against many token positions, so arithmetic intensity is high and the operation is compute-bound. Prefill time scales with prompt length and with model FLOPs.

Decode produces one token at a time. Every weight is loaded and multiplied against a single position per sequence, so arithmetic intensity is roughly the batch size, and at small batch the operation is entirely memory-bound. Decode time per token is governed by

which contains no FLOP count at all. A 13B model in fp16 is 26 GB; on a device with 2 TB/s of bandwidth the floor is about 13 ms per token, or 77 tokens per second, before anything else is considered. This calculation should come before any architecture discussion, because it bounds what is achievable.

Which design moves affect which term

Reducing bytes per weight attacks decode directly. Weight-only 4-bit quantisation cuts the dominant term by roughly four, and this is why it is the first move in almost every latency project.

Reducing KV cache bytes matters as context grows and as concurrency rises. Grouped-query attention cutting KV heads from 32 to 8 reduces cache traffic and size by four; multi-head latent attention compresses further. At long context the cache overtakes the weights as the larger term, and past that point cache design dominates.

Reducing layer count attacks the fixed per-layer overhead: kernel launches, synchronisations, and normalisation reductions that do not batch away. At batch size one on a small model this overhead can be a third of decode time, which is why shallow-wide beats deep-narrow for latency at equal parameters.

Reducing active parameters through a mixture of experts cuts prefill FLOPs substantially and cuts decode time much less, because all expert weights must still be resident and the routing brings its own memory traffic. MoE is a training-efficiency and capability move whose inference benefit is smaller than the active-parameter ratio suggests.

Working backwards

Start with the budget, compute the byte ceiling from bandwidth, subtract the KV cache at the target context and concurrency, and what remains is the weight budget in bytes. Divide by the bits per weight you are willing to ship and that is the parameter count. Only then choose depth, width and attention configuration to fit it, using the loss-versus-shape insensitivity discussed under depth and width to spend the remaining freedom on whatever helps throughput.

This ordering is the point. Choosing a parameter count first and hoping the latency works out is the standard failure, and it usually ends with a model that has to be quantised and pruned into shape afterwards, losing more quality than a correctly sized model would have.

When it breaks

Concurrency changes the regime. At batch 64 decode is no longer purely memory-bound, because one weight load serves 64 sequences. A design optimised for single-user latency can be badly suboptimal for throughput at high concurrency, and vice versa. The budget must specify both, or the design optimises for a workload that does not exist.

The cache eats the batch. KV cache per sequence grows linearly with context, so at long context the number of concurrent sequences that fit is set by cache size, not by weights. A model that is fast per token and allows only eight concurrent users can be more expensive per request than a slower one allowing 64.

Speculative decoding changes the arithmetic. Verifying several draft tokens in one forward pass raises arithmetic intensity, moving decode toward compute-bound and partly invalidating the memory-bound analysis. A model designed for speculative decoding wants a good small draft model in the same family, which is an architecture decision made at pretraining time.

Bandwidth numbers are peak numbers. Achieved bandwidth on real kernels is typically 70 to 85 percent of specification, so the theoretical floor is optimistic by that margin. Budgets computed from peak figures should carry the discount explicitly rather than discovering it in benchmarking.