Disaggregated Prefill and Decode Serving
Why prefill and decode want opposite hardware and parallelism, how splitting them across separate GPU pools raises goodput, and what the KV cache transfer costs.
A single LLM request runs two workloads that have almost nothing in common. Prefill processes the entire prompt in one pass: thousands of tokens, large matrix multiplications, compute-bound, GPU arithmetic units saturated. Decode generates one token at a time: a tiny matrix multiply against the full weight matrix and the growing KV cache, memory-bandwidth-bound, arithmetic units mostly idle while HBM is the bottleneck.
Colocating them on the same GPU, which is what every serving system did through 2023, forces one set of decisions on both. That is the problem disaggregation solves.
The interference
In a colocated server, a long prefill and a stream of decodes contend for the same scheduler. When a 16,000-token prompt arrives, it occupies the GPU for hundreds of milliseconds, and every request currently streaming waits. Their TPOT spikes; the user sees the text stop mid-sentence. Continuous batching improved utilisation but did not remove this, because prefill and decode still share an iteration.
Two responses exist, and they are complements rather than rivals:
- Chunked prefill keeps them together but slices long prefills into pieces small enough to co-schedule with decodes, so no decode iteration stalls (Sarathi-Serve, arXiv:2403.02310).
- Disaggregation separates them onto different GPU pools entirely.
graph LR R[Request] --> P[Prefill pool<br/>compute-bound<br/>TP-heavy] P -->|KV cache transfer| D[Decode pool<br/>bandwidth-bound<br/>large batches] D --> S[Stream to client]
What separation buys
Once the phases are on separate pools, every configuration decision can be made twice.
| Dimension | Prefill pool | Decode pool |
|---|---|---|
| Bottleneck | compute (FLOPs) | memory bandwidth and KV capacity |
| Batching | small batches, already saturated | large batches, more is better |
| Parallelism | tensor parallelism to cut TTFT | often lower TP, more replicas |
| Scaling signal | prompt token arrival rate | concurrent active sequences |
| Good hardware fit | newest, highest-FLOP parts | high-HBM parts, older parts viable |
The pools also scale independently. A summarisation workload with 20,000-token inputs and 200-token outputs is prefill-heavy; a chat workload with 300-token inputs and 800-token outputs is decode-heavy. Colocated, you provision for the sum of the peaks. Disaggregated, you provision each for its own. DistServe reported serving 7.4x more requests, or meeting a 12.6x tighter SLO, while keeping over 90 percent of requests inside their latency constraints (Zhong et al., OSDI 2024).
The cost: moving the KV cache
Disaggregation introduces work that did not exist before. The prefill worker produces the KV cache for the whole prompt, and the decode worker needs it. That is a transfer of, for a 70B-class model, on the order of hundreds of megabytes to several gigabytes for a long prompt, on the critical path of every request.
This is why the topology matters more than the idea. Over NVLink or InfiniBand with RDMA the transfer can overlap with the tail of prefill and largely disappear; over ordinary Ethernet it can cost more than the interference it removed. The engineering has consolidated around dedicated transfer layers: NVIDIA's NIXL in Dynamo, adopted by llm-d, TensorRT-LLM, SGLang and vLLM, and Mooncake's KVCache-centric design that pools idle CPU DRAM and SSD across the cluster as a shared cache tier, which won Best Paper at FAST 2025 (Qin et al., arXiv:2407.00079).
When it breaks
- Small deployments lose. Below roughly a handful of GPUs, splitting pools means each is too small to batch well, and static partitioning wastes capacity that colocation would have shared. Chunked prefill is the better answer at that scale.
- Traffic mix shifts. The prefill-to-decode ratio is set by prompt and output lengths. A workload change moves the optimal split, and a fixed partition is now wrong in one direction. Systems that rebalance dynamically exist but add a control plane.
- The transfer is a new failure domain. A prefill worker dying mid-transfer, or a network partition between pools, breaks requests in a way a single colocated worker never could.
- Complexity is real. Two pools, a transfer layer, a router that must track both, and metrics that no longer decompose per node. Adopt it when the goodput math justifies the operational surface, not before.
10 flashcards for this concept
Click a card to reveal the answer.