A document assistant sends a 30000-token prompt and streams about 400 tokens back. Time to first token is 4 seconds and the remaining 400 tokens arrive over the following 6 seconds. Which change most reduces the 4 seconds?

What is being tested

Whether you know that generation has two phases with different costs and different levers, and can tell which phase a given number belongs to. Teams that do not make this distinction optimise the wrong half and see no change.

The mechanism

• Prefill processes the whole prompt in one pass to build the key-value cache. It is parallel across tokens and compute-bound, so its duration scales with prompt length. 30,000 tokens taking about 4 seconds is prefill.
• Decode then produces output one token at a time, each step reading the entire model's weights and the growing key-value cache from memory. It is sequential and memory-bandwidth-bound, so its duration scales with output length and with how much bandwidth the batch leaves for you. 400 tokens in 6 seconds is about 15 ms per token, which is a decode figure.

So time to first token is a prompt-length problem. Prompt caching attacks it directly: if the first 29,000 tokens are the same instructions and documents on every request, the serving stack can retain their key-value cache and process only the new suffix, which turns most of the prefill into a lookup. Published pricing for cached prompt tokens is typically a fraction of the uncached rate, so the cost falls along with the latency.

Why the other options fail

• Lowering the output limit changes total response time, not time to first token. The first token arrives before any output length matters.
• Network latency is tens of milliseconds between regions. It cannot account for 4 seconds, and this is the most common misdiagnosis because it is the easiest thing to blame.
• Raising the batch size improves tokens per second across all users and usually makes a single request slower, because more sequences share the same memory bandwidth during decode. It is a throughput lever, and the question asks about latency.

When this is the wrong lever

When there is no shared prefix. If every request's prompt is genuinely unique - a different document each time, with no common instruction block - there is nothing to cache and the only way to cut prefill is to send fewer tokens: retrieve the relevant passages instead of stuffing the whole document, and order the prompt so the stable part comes first so that future caching becomes possible. Prompt structure is a latency decision, which is why it belongs in the design rather than in prompt iteration. If the 4 seconds is unavoidable, the remaining move is perceptual: stream a retrieval or plan step first so the user sees progress while prefill runs.