An LLM chat product serves millions of multi-turn conversations. How do KV-cache reuse, prefix caching, continuous batching and session affinity change the architecture, and what breaks when a session lands on a different GPU?

Why the KV cache dominates the architecture

Transformer inference has two phases with completely different characteristics. Prefill processes the whole prompt at once — compute-bound, highly parallel. Decode generates one token at a time, each step attending over all previous tokens — memory-bandwidth-bound, and requiring the key/value tensors for every prior token to be held in GPU memory.

That held state is the KV cache, and its size is roughly layers × heads × head-dimension × sequence length × 2 × precision-bytes. For a long multi-turn conversation it is large, per session, and resident for the duration of the request.

GPU memory therefore limits concurrency, not compute — which inverts the usual intuition and is the single most important fact about serving these models.

KV-cache reuse across turns

In a chat product, turn N+1's prompt is turn N's conversation plus new text. The KV cache for the shared prefix is identical and can be reused rather than recomputed.

Without reuse, every turn re-prefills the entire history: a 20-turn conversation re-processes an ever-growing prompt each time, so cost grows quadratically with conversation length. With reuse, only the new tokens are prefilled, and cost grows linearly.

This is the difference between a viable consumer product and an unaffordable one, and it is why Character.AI's published work on serving efficiency centres on cache size and reuse.

The techniques that make it work

• Prefix caching, generalised: any shared prefix — a long system prompt, a character definition, a retrieved document set used by many requests — is computed once and reused across sessions. For products with large fixed system prompts this is an enormous saving, because the shared portion may dominate.
• Multi-query and grouped-query attention, which share key/value heads across query heads and reduce KV cache size by a large factor, directly increasing the number of concurrent sessions per GPU. This is a model architecture choice made for serving economics.
• Quantising the cache to lower precision, trading a little quality for substantially more concurrency.
• Paged attention, allocating the cache in fixed-size blocks rather than contiguously — which removes the fragmentation that otherwise wastes a large fraction of GPU memory and allows blocks to be shared between sessions with a common prefix.
• Continuous batching: rather than waiting for a batch to complete, finished sequences leave the batch and new requests join at each step. This raises utilisation dramatically, because sequence lengths vary enormously and static batching wastes the difference.
• Disaggregating prefill and decode onto separate pools, since one is compute-bound and the other memory-bandwidth-bound, allowing each to be provisioned and scaled independently.

Session affinity, and what breaks without it

Cache reuse requires the session to return to the GPU holding its cache. So the router must be session-aware, which introduces the properties of a stateful system into what everyone would prefer to be stateless:

• A routing failure means a cache miss, and the next turn re-prefills the entire history — a latency spike proportional to conversation length, precisely for the users with the longest and most valuable conversations.
• Load balancing becomes constrained. Requests cannot simply go to the least-loaded instance, so hot instances develop and a naive rebalancing destroys caches.
• Node loss loses the caches of every session on it, producing a burst of expensive re-prefills at exactly the moment capacity has been reduced — a self-amplifying failure.
• Deployments must drain gently, since restarting an instance evicts every session's cache.

Mitigations: a tiered cache that spills to CPU memory or fast local storage, so a miss is a slow reload rather than a full recomputation; admission control on re-prefill, so a wave of them cannot saturate the fleet; and accepting graceful degradation — dropping older turns from the context — as a defined behaviour rather than an accident.