Prefill-Decode Asymmetry
also called Two-Phase Inference Cost, TTFT versus ITL
The fact that generating a response has two phases with opposite bottlenecks - a compute-bound pass over the whole prompt, then a memory-bandwidth-bound loop producing one token at a time - so each phase responds to different levers and optimising the wrong one changes nothing.
A document assistant sends a 30,000-token prompt and streams 400 tokens back. The first token arrives after 4 seconds; the remaining 400 take 6 more. Those two numbers come from two different machines, in effect, and no single optimisation improves both.
Prefill processes the entire prompt in one pass to build the key-value cache. All tokens are available at once, so the work is a dense matrix computation that saturates arithmetic units: it is compute-bound and its duration scales with prompt length. Decode then produces output tokens one at a time, and each step must read the model's weights and the whole growing cache from memory to produce a single token. It is memory-bandwidth-bound and its duration scales with output length.
The asymmetry is the whole reason time to first token and inter-token latency are reported separately, and why a team that optimises "latency" without splitting them usually ends up with no measurable change.
Why it matters
The two phases have disjoint levers. Time to first token responds to prompt length, prefix caching and prompt structure. Inter-token latency responds to model size, quantisation, memory bandwidth, batch composition and speculative decoding. Pulling a decode lever to fix a prefill number is the most common wasted optimisation in LLM applications.
It also explains a counter-intuitive operational fact: raising batch size improves tokens per second across all users and usually makes any single user slower, because concurrent sequences share the same memory bandwidth during decode. Throughput and latency are in tension, and the tension lives entirely in the decode phase.
Finally it sets the economics. Input tokens are typically cheaper per token than output tokens, and cached input tokens cheaper still, which is a direct consequence of prefill being parallel and cacheable while decode is neither.
Implementation patterns
- Measure the two separately. Time to first token and inter-token latency, as distinct service level indicators. An end-to-end number hides which half is wrong.
- Structure the prompt so the stable part comes first. Prefix caching only works on a prefix, so system instructions and shared documents go at the front and the variable part at the end. This is a design decision, not a prompt-engineering detail.
- Cut prompt tokens rather than hoping for faster prefill: retrieve the relevant passages instead of stuffing whole documents, and drop conversation history that no longer affects the answer.
- For decode latency, reach for a smaller or quantised model, more bandwidth, or speculative decoding at low concurrency - and expect none of them to move time to first token.
- Stream something early when prefill is irreducible. A retrieval step or a plan rendered while prefill runs changes the perceived latency without changing the number.
- Separate interactive and batch traffic onto different serving configurations, because the right batch size for throughput is the wrong one for latency.
Industry example
Hosted model providers price the two phases differently - input, cached input and output tokens at distinct rates, with cached input typically a fraction of uncached - which is the asymmetry expressed as a tariff. Serving systems built since 2023 are organised around it as well: continuous batching exists to keep decode busy across requests, chunked prefill exists to stop a long prompt's compute-bound pass from stalling other users' decode steps, and prefix caching exists because prefill over a repeated prefix is pure waste. Each of those features targets exactly one phase.
Failure scenarios
- Optimising the wrong phase. Weeks spent on model quantisation to fix a 4-second time to first token that was caused by a 30,000-token prompt.
- Blaming the network. Four seconds attributed to connectivity when inter-region latency is tens of milliseconds.
- Prefix broken by a timestamp. A dynamic value placed at the top of the prompt invalidates the cache on every request, so the cache hit ratio silently collapses and cost rises with no code change anyone remembers.
- Batch size raised for throughput, with interactive users becoming noticeably slower and the two effects never connected because only an aggregate latency figure is charted.
- Context growth in agent loops. Prefill is paid again on every turn over an ever-longer prompt, so latency and cost grow superlinearly across a task while each individual call looks reasonable.
- An output token limit raised to fix time to first token, which cannot work by construction.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Large batches | Highest tokens per second per accelerator | Slower individual responses; latency variance under load |
| Small batches or dedicated capacity | Low and predictable latency | Poor utilisation and higher cost per token |
| Aggressive prefix caching | Large reduction in time to first token and input cost | Prompt layout becomes a constraint; cache invalidation becomes a failure mode to monitor |
When not to use it
The distinction stops being useful when prompts and outputs are both short - a classification call with 200 tokens in and 5 out is dominated by queueing and network, and splitting the phases measures noise.
It is also not the right frame for embedding workloads, which have no decode phase at all: they are prefill-only, so their cost and latency scale purely with input tokens and the whole asymmetry collapses.
Interview question
Q: Your assistant's p95 end-to-end latency is 12 seconds and users say it feels slow to start. What do you measure, and what are the first three changes you would consider?
What a strong answer covers: splitting the measurement into time to first token and inter-token latency before proposing anything · recognising a long prefill as a prompt-length problem and naming prefix caching, prompt reordering so the stable block is first, and retrieval instead of document stuffing · naming the decode levers separately and saying explicitly that they will not help time to first token · the batch-size tension between throughput and single-request latency, and separating interactive from batch traffic · perceived-latency work such as streaming an early step when prefill is irreducible · and the cost consequence, since cached input tokens are billed at a fraction of uncached ones.
Quick check
Quiz: Why does raising the batch size improve throughput and worsen single-request latency? Because decode is memory-bandwidth-bound and concurrent sequences share that bandwidth, so each request's tokens arrive more slowly while the accelerator produces more tokens in total.
Flashcard: Which phase owns time to first token, and which lever applies? - Prefill, which is compute-bound and scales with prompt length; the levers are prefix caching, prompt reordering so the stable part comes first, and sending fewer tokens - not model size, not the network, not the output limit.