Latency, Streaming and Perceived Speed
Why time to first token dominates perceived speed, how streaming changes what users tolerate, and the interaction costs that come with it.
Generation is slow in a way that ordinary software is not, and the response has been to change what the user waits for rather than to make the waiting shorter. Understanding which part of the latency users actually perceive is what makes that trade work.
The two numbers
Time to first token is how long the user stares at nothing. It is dominated by prefill, which scales with prompt length, plus queueing, plus any retrieval that runs first. This is the number users experience as responsiveness, and the perceptual thresholds are the familiar ones: under about 100 ms feels instant, under a second keeps attention, and beyond a few seconds requires something to look at or the user disengages.
Inter-token latency determines how fast the response unfolds once it starts. Above roughly 20 tokens per second, output outpaces comfortable reading, so further improvement is largely imperceptible for text a human reads. This is why optimising decode throughput past a point buys throughput and not experience.
Optimising the wrong one is common. A team that halves total generation time by speeding up decode has improved a number users had stopped noticing, while a 400 ms reduction in time to first token is felt on every request.
What streaming buys and costs
Streaming converts a long wait into an immediate start, and users tolerate substantially longer total durations when output is arriving. It also lets them begin reading, evaluating and deciding to stop before generation completes.
The costs are real. Output cannot be validated before display, so schema checks, safety filters and formatting all have to work incrementally or be abandoned. Rendering structured output while it is incomplete produces visible half-formed markup. Users read partial output and act on it before the qualification arrives in the next sentence. And a stopped generation leaves a partial response whose status has to be communicated.
Filling the gap before the first token
Where time to first token cannot be reduced, showing what is happening beats a spinner. Displaying the retrieval step, the tools being called, or the plan being formed gives the user something to read during the wait, sets expectations about duration, and doubles as the transparency that makes verification possible later.
For agent workflows this is the difference between an opaque thirty-second pause and a legible sequence, and it changes the tolerable duration substantially.
When it breaks
Progress indication that does not correspond to progress erodes trust. A fake progress bar or an invented step list is discovered quickly, and after that no status display is believed. Report the real steps or report nothing.
Perceived speed masks a cost problem. Streaming makes a slow, expensive generation feel acceptable, which removes the pressure to shorten it. The token bill does not care how it felt.
Long agent runs need a different interaction model entirely. Beyond roughly a minute, synchronous waiting stops working regardless of what is displayed. The design that fits is asynchronous: acknowledge, work in the background, notify on completion, which is a substantially different product than a chat box.
Latency variance matters as much as the mean. A system that usually responds in one second and sometimes in fifteen feels unreliable in a way a consistent three-second system does not. Tail latency is the experience, and p50 reporting hides it.
12 flashcards for this concept
Click a card to reveal the answer.