Prefill vs Decode

A 7B-parameter model on an A100 can saturate the GPU's tensor cores during prefill, then immediately become memory-bandwidth-limited once decode begins. These are not just two stages of the same job; they are two different workloads wearing the same hardware.

The Two Phases

Every autoregressive LLM inference call passes through exactly two phases.

Prefill takes the full prompt (P tokens) and processes all of them simultaneously in a single forward pass. Every token attends to every other token in the prompt. The attention computation is O(P²) in time and O(P) in KV cache memory. Because you are doing a large matrix multiplication - (batch × heads × P × d_head) × (d_head × d_model) - you are keeping the GPU's arithmetic units busy. This is a compute-bound workload.

Decode takes the last generated token and runs one forward pass to produce the next token. At each step, only a single new row is added to the KV cache; the rest is just read back from memory. The matrix multiplications shrink to (batch × heads × 1 × d_head), which is a vector-matrix product. The GPU spends most of its cycle budget waiting for weight tensors and KV cache to arrive from HBM, not performing arithmetic. This is a memory-bandwidth-bound workload.

The roofline model makes this crisp. Arithmetic intensity (FLOPs per byte transferred) determines which bound applies:

Arithmetic intensity = FLOPs / bytes_read

Prefill:  FLOPs ∝ 2 × P × d_model²   (large matrix product)
          bytes ∝ d_model²             (load weights once)
          → intensity grows with P; sits above the roofline ridge

Decode:   FLOPs ∝ 2 × d_model²        (one token - vector×matrix)
          bytes ∝ d_model²             (still load the full weights)
          → intensity ≈ 1 FLOPs/byte;  far below the ridge

On an A100 SXM4, peak FP16 compute is ~312 TFLOP/s and peak HBM2e bandwidth is ~2 TB/s. The ridge point sits at roughly 156 FLOP/byte. Single-token decode at batch size 1 achieves ~1-2 FLOP/byte, meaning you are using perhaps 1-2% of peak compute but potentially 60-80% of peak bandwidth. You are not "wasting" compute; you are hitting a different ceiling entirely.

Why This Matters for Serving Systems

Mixing prefill and decode in the same batch creates interference. A long prompt arriving mid-decode causes a "prefill stall": decode steps pause while the prefill request monopolises the GPU for tens or hundreds of milliseconds, spiking time-to-first-token (TTFT) for all inflight requests.

Continuous batching (as in Orca / vLLM) mitigates head-of-line blocking by scheduling at the token level, but the CPU/GPU scheduling overhead and the prefill stall problem remain when prompt lengths are long.

Chunked prefill (introduced in Sarathi-Serve) addresses the stall by splitting a large prefill into fixed-size chunks - say 512 tokens at a time - and interleaving them with decode steps. A large prompt no longer blocks decode for hundreds of milliseconds; it yields the GPU after each chunk. The tradeoff is that chunked prefill increases time-to-first-token for the affected request (more scheduling rounds), while improving inter-token latency (ITL) for already-decoding requests.

Prefill-decode disaggregation takes a more radical position: run prefill and decode on separate machines (or at least separate GPU pools). Splitwise (2023) showed that prompt computation and token generation have so different resource profiles that using identical high-end GPUs for both wastes money. Decode does not need the latest tensor-core generation; an older GPU with high HBM bandwidth is often a better fit. Mooncake (2024), Kimi's production inference system, operationalises this by separating prefill clusters and decoding clusters, transferring KV cache over a fast interconnect after each prefill completes.

The KV cache transfer is the practical bottleneck in disaggregation: a 7B model at FP16 with sequence length 4096, 32 layers, and 32 heads has a KV cache of roughly 2 × 32 × 4096 × 128 × 2 bytes ≈ 1 GB per request. At 100 GB/s interconnect bandwidth (NVLink within a node), that transfer takes ~10 ms - acceptable, but not free. At rack-scale Ethernet (25-100 Gbps), it can exceed the decode latency budget.

Batch Size Changes the Arithmetic

The compute vs. bandwidth picture shifts with batch size. At large batch sizes, decode regains arithmetic intensity because weight bytes are amortised across many simultaneous requests:

This is why throughput-oriented serving runs large decode batches: it transforms a bandwidth-bound problem into a compute-bound one, maximising GPU utilisation. The flip side is increased queuing latency. Latency-sensitive applications (chat, coding assistants) need small batch sizes and thus accept lower hardware utilisation.

Prefill does not have this dependency on batch size in the same way; it is inherently compute-bound as soon as prompt length is more than a handful of tokens.

Practical Knobs

Most production serving frameworks expose parameters that map directly to this duality:

• max_num_batched_tokens - caps the total tokens (prefill + decode) per scheduling step; controls GPU memory pressure.
• chunked_prefill_size / max_prefill_tokens - the chunk size for interleaved prefill; smaller values improve decode ITL, larger values reduce TTFT for prefill requests.
• prefill_chunk_size in disaggregated deployments - determines how often KV cache is flushed from the prefill instance to the decode instance.

Profiling tools like nsys or torch.profiler will show prefill steps as dense compute kernels (cuBLAS GEMM) and decode steps as bandwidth-saturating memory transfers (with short, serial GEMM calls).

When It Falls Down

Short prompts annihilate the prefill/decode distinction. A 10-token prompt behaves almost like decode - low P means low arithmetic intensity. Optimisations tuned for long-prompt prefill (e.g., flash attention with large tile sizes) may actually hurt throughput on short prompts.

Speculative decoding blurs the phases. Draft models generate multiple tokens per step; each verification step resembles a short prefill for the target model. Profiling tools that assume alternating prefill/decode steps give misleading readings.

Disaggregation breaks down at small scale. If you have only two or four GPUs, running separate prefill and decode pools wastes one and leaves the other idle during off-peak periods. The efficiency gains only materialise at cluster scale with steady, mixed-length traffic.

Very long contexts shift decode toward memory-bound for a different reason. As the KV cache grows, each decode step must read back more KV entries from HBM. The bandwidth bottleneck is now the KV cache, not the weights. This changes the optimal batching strategy: you want fewer, longer sequences per batch rather than many short ones.

Chunked prefill hurts single-request TTFT. If a user submits a 32k-token document and chunked_prefill_size is 512, the model needs 64 scheduling rounds before producing the first token. Choosing the right chunk size is a per-deployment tuning problem, not a one-size-fits-all setting.

Further Reading

• Splitwise: Efficient generative LLM inference using phase splitting - original disaggregation paper characterising compute vs. memory profiles per phase.
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving - production-scale disaggregated serving separating prefill and decode clusters.
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve - introduces chunked prefill and stall-free scheduling.
• Efficient Memory Management for Large Language Model Serving with PagedAttention - vLLM's continuous batching foundation; context for understanding the scheduling layer that sits above prefill/decode.