KV Cache

Transformer inference has two phases. Prefill processes the entire prompt in parallel and is compute-bound. Decode produces one token at a time, and each step attends over every preceding token — which requires the key and value projections for all of them to remain in GPU memory.

That resident state is the KV cache. Its size is approximately layers × KV heads × head dimension × sequence length × 2 × bytes per element, per sequence, and for long multi-turn conversations it is large.

The consequence that reshapes the architecture: GPU memory, not compute, limits how many sessions a server can handle concurrently.

Why it matters

Recomputing the cache is the dominant avoidable cost in a chat product. Turn N+1's prompt is turn N's conversation plus new text, so without reuse each turn re-prefills the entire growing history and cost grows quadratically with conversation length. With reuse, only the new tokens are processed and cost grows linearly.

The second consequence is architectural rather than economic. Cache reuse requires the session to return to the machine holding its cache, which makes an inference tier stateful — with routing, rebalancing, draining and failure behaviour that a stateless service would not have.

Implementation patterns

• Cross-turn reuse within a session, prefilling only the new tokens.
• Prefix caching across sessions for shared content — a long system prompt, a persona definition, a retrieved document set. Where the fixed prefix dominates the prompt, this is the largest single saving available.
• Grouped-query or multi-query attention, sharing KV heads across query heads to shrink the cache by a large factor. This is a model architecture decision made for serving economics, and it directly increases concurrent sessions per GPU.
• Cache quantisation to lower precision, trading a little quality for substantially more concurrency.
• Paged attention — allocating in fixed-size blocks rather than contiguously — which removes fragmentation that otherwise wastes a large share of GPU memory, and allows blocks to be shared between sequences with a common prefix.
• A tiered cache, spilling to CPU memory or fast local storage, so an affinity miss is a slow reload rather than a full recomputation.
• Session-aware routing with graceful draining, and admission control on re-prefill so a wave of misses cannot saturate the fleet.
• Defined degradation: dropping older turns from the context under pressure, as a designed behaviour rather than an accident.

Industry example

Character.AI's published work on inference efficiency centres on exactly these mechanisms — reducing KV cache size through attention architecture choices and reusing cache across conversation turns — because the product serves very large volumes of long multi-turn conversations at consumer price points, where the quadratic recomputation cost would be prohibitive.

The same techniques underpin every high-throughput serving stack: paged attention and continuous batching are now standard in open inference servers, and prefix caching is a headline feature of commercial model APIs precisely because shared system prompts are ubiquitous in real applications.

Failure scenarios

• No cross-turn reuse, producing quadratic cost growth that appears as an unexplained bill.
• Routing that ignores affinity, so every turn is a cache miss and long conversations get the worst latency.
• Node loss triggering mass re-prefill at the moment capacity has been reduced — a self-amplifying failure.
• Restarting instances during a deploy without draining, evicting every session's cache simultaneously.
• Memory fragmentation from contiguous allocation, leaving a large fraction of GPU memory unusable.
• Admission control based on request count, which ignores the resource that actually binds.
• Unbounded context growth, where a long-running session's cache slowly consumes the memory that would have served many other users.
• Cache retained across users, which is a data-leakage risk if prefix sharing is implemented carelessly.

Trade-offs

Cache reuse trades statelessness for cost, and statelessness is genuinely valuable: simple load balancing, trivial failover, easy scaling. Affinity gives all of that up, and the operational complexity is real.

Shrinking the cache through attention architecture or quantisation trades model quality for concurrency, in amounts that must be measured rather than assumed — and the decision is usually made at model-training time, long before the serving team is consulted.

The trade is operational simplicity and some quality in exchange for an order-of-magnitude difference in serving economics. For a low-volume internal tool, none of this is worth doing. For a consumer product with long conversations, the naive stateless design is not merely inefficient — it is unaffordable, which is why this is one of the few areas where the optimisation is not optional.

Interview question

"Our chat product's inference bill grows faster than our user count and we cannot explain it. Tell me what you would look at first, then design the fix — and tell me what we give up operationally by implementing it."