Chunked Prefill

Prefill and decode compete for the same accelerator. Prefill is a large, compute-heavy operation over the whole prompt; decode is a small step producing one token per sequence. When a request with a very long context arrives, its prefill occupies the GPU for a substantial period.

Every other sequence's decode stops during that time. Users mid-conversation experience a stall proportional to somebody else's prompt length — a fairness failure invisible in aggregate throughput metrics, and highly visible to the affected users.

Chunked prefill splits the long prefill into pieces and interleaves them with decode steps for the other sequences, so the long request progresses without monopolising the device.

Why it matters

It is the mechanism that makes mixed workloads viable on shared inference capacity. Without it, an inference fleet serving both interactive chat and long-document processing must either separate them physically — which fragments capacity and reduces utilisation — or accept that document requests periodically stall conversations.

The effect on the metric that users perceive is direct: inter-token latency becomes bounded rather than occasionally catastrophic. Aggregate throughput may fall slightly, because prefill is most efficient as one large operation, and the tail latency improvement is worth far more than the throughput lost.

Implementation patterns

• Chunk size tuned to the acceptable decode stall. Smaller chunks mean smoother decode and slightly lower prefill efficiency; the parameter is a direct latency/throughput dial and should be set from an SLO.
• Combined with continuous batching, so each iteration mixes decode steps for established sequences with one prefill chunk for an arriving one.
• Priority-aware chunking: lower-priority long-context requests get smaller or less frequent chunks, so interactive traffic is protected explicitly rather than incidentally.
• Paired with memory-based admission control, since chunking addresses the latency problem and not the memory problem — a long context still requires its full KV cache.
• Separate prefill and decode pools as the alternative or complement, so a burst of long prompts saturates one pool without affecting the other's latency.
• Measured through inter-token latency at the tail, not average throughput, since the whole point is the distribution.
• A cap on maximum context per priority class, so that even chunked, a single request cannot occupy an instance indefinitely.

Industry example

Chunked prefill is a standard feature of modern open inference servers and is documented as one of the primary mechanisms for meeting interactive latency targets on fleets that also process long-context requests. Its adoption tracked the growth of context windows: at 4,000 tokens the stall was tolerable, and at 100,000-plus tokens a single unchunked prefill became long enough to violate every interactive SLO on the machine.

It sits alongside continuous batching and paged attention as one of the three scheduling advances that made high-utilisation multi-tenant inference practical, each addressing a different waste: batching addresses idle compute between requests, paging addresses fragmented memory, and chunking addresses head-of-line blocking.

Failure scenarios

• No chunking on a mixed fleet, producing periodic multi-second stalls for every active conversation.
• Chunks too large, so the mechanism exists and the stall remains perceptible.
• Chunks too small, wasting prefill efficiency for no further latency benefit.
• Chunking without memory admission control, so latency is smooth right up until the point where memory is exhausted and requests fail.
• No priority awareness, so a batch job's long prompt is chunked identically to an interactive one and receives capacity it does not need.
• Measuring only throughput, which chunking slightly reduces, leading to the feature being disabled by someone optimising the wrong metric.

Trade-offs

Chunked prefill reduces raw prefill throughput — a single large matrix operation is more efficient than several smaller ones — and adds scheduling complexity to the serving loop.

It also does not reduce total work or memory pressure. A long-context request still consumes the same compute and the same KV cache; chunking only changes when the compute happens. Teams sometimes adopt it expecting a capacity improvement and are surprised that throughput falls slightly.

The trade is a small amount of aggregate throughput in exchange for bounded inter-token latency under a mixed workload. For a fleet serving only long batch jobs it is unnecessary. For any fleet where a human is waiting on one request while another processes a large document, it is the difference between a usable product and one that intermittently freezes.

Interview question

"Our chat latency is fine on average and users complain about freezes. We also run a document-summarisation API on the same fleet. Tell me what is happening, what you would change, and what metric would prove the fix worked."