Serving SLOs: TTFT, TPOT and Goodput
Why tokens per second is the wrong number to optimise, how TTFT and TPOT split the latency budget, and what goodput measures that throughput hides.
Two deployments serve the same model on the same hardware. The first reports 4,200 tokens per second, the second 2,600. The first is unusable for a chat product, because its median request waits 6 seconds before the first token appears. Throughput is a property of the server; latency is a property of the request. Optimising the former alone reliably produces a system that is fast on paper and slow to use.
LLM serving needs at least three numbers, and they measure different phases of the same request.
The three that matter
TTFT (time to first token) covers queueing, scheduling and the prefill forward pass over the whole prompt. It scales with prompt length and with how loaded the queue is. It is what the user experiences as "did it hear me".
TPOT (time per output token), also called inter-token latency, is the steady-state gap between tokens during decode. It scales with batch size and memory bandwidth, not with prompt length. It is what the user experiences as reading speed. A useful anchor: comfortable reading is roughly 30 to 50 ms per token, so a TPOT above about 80 ms feels laboured regardless of how fast the first token arrived.
End-to-end latency is \(\text{TTFT} + \text{TPOT} \times (\text{output tokens} - 1)\). For a 500-token answer, TPOT dominates by an order of magnitude, which is why chat products tune TPOT and batch APIs mostly do not care about it.
Different products weight these differently:
| Workload | Binding constraint |
|---|---|
| Interactive chat | TTFT under ~500 ms, TPOT under ~50 ms |
| Voice agent | TTFT under ~300 ms, TPOT below speech rate |
| Coding autocomplete | TTFT is nearly everything |
| Agent tool loop | end-to-end per step, TTFT paid on every step |
| Batch summarisation | throughput and cost only |
Goodput, and why it replaces throughput
The metric that makes SLOs comparable is goodput: the request rate a system sustains while meeting both its TTFT and TPOT targets. A server that answers 100 requests per second with 40 percent of them violating TTFT has a goodput of 60, and the DistServe authors made this the objective function rather than a reporting detail (Zhong et al., OSDI 2024, arXiv:2401.09670).
This reframing matters because the two phases fight each other. Batching more requests raises throughput and raises TPOT for everyone in the batch. Admitting a long prompt lets a prefill pass block the decode loop, spiking TPOT for requests already streaming. Sarathi-Serve named this the throughput-latency tradeoff directly and attacked it by splitting long prefills into chunks that ride alongside decodes, so no decode iteration stalls behind a full prefill (Agrawal et al., OSDI 2024, arXiv:2403.02310).
Measuring honestly
- Report percentiles, not means. p50 TTFT tells you the system works; p99 tells you whether users trust it. Queueing effects live entirely in the tail.
- Fix the input distribution. TTFT at 200-token prompts and at 20,000-token prompts are different systems. A benchmark that does not state prompt and output length distributions is not reproducible.
- Measure under load, not in isolation. Single-request latency measures the model. Latency at target QPS measures the scheduler, and the scheduler is what breaks.
- Streaming hides and reveals. Streaming makes TTFT the perceived latency, which is why it is worth doing, and it also means a TPOT regression shows up as visible stutter rather than a slower total time.
When it breaks
The most common failure is optimising a metric the product does not have. Turning on aggressive batching to raise tokens per second raises TPOT; enabling speculative decoding cuts TPOT at low load but wastes compute and can hurt goodput when the server is saturated, because rejected draft tokens consume the batch slots real requests needed.
The second is treating GPU utilisation as a health signal. Continuous batching keeps utilisation near 100 percent by design, so it stays pinned whether the system is comfortable or collapsing. Queue depth, KV cache occupancy and SLO attainment are the signals that actually move before users notice.
10 flashcards for this concept
Click a card to reveal the answer.