Cross-Layer KV Sharing

Multi-query attention and grouped-query attention cut the KV cache along the head axis, and by 2023 everyone had taken that discount. GQA with 8 key-value groups instead of 64 heads is an 8x reduction, and it is close to free (Ainslie et al., 2023, GQA, arXiv:2305.13245, building on Shazeer, 2019, Fast Transformer Decoding, arXiv:1911.02150). Sliding-window attention cuts along the sequence axis. Both are now standard, and the cache is still the thing that decides your batch size at 128k context.

There is a third axis nobody was using: depth. A 32-layer model stores 32 independent key-value tensors per token, one per layer, and there is no law saying each layer needs its own.

Two layouts that mean different things

Cross-Layer Attention (CLA) ties the K and V projections of adjacent layers. Layer \(2i+1\) computes its own queries but reads the keys and values produced by layer \(2i\). Sharing factor 2 halves the number of cached tensors; sharing factor 3 divides by three. Evaluated at 1B and 3B scale against tuned baselines, CLA on top of MQA delivers roughly another 2x KV reduction at close to unchanged accuracy, which moves the memory-accuracy Pareto frontier rather than trading along it (Brandon et al., 2024, Reducing Transformer Key-Value Cache Size with Cross-Layer Attention, arXiv:2405.12981).

YOCO restructures the model instead of tying pairs. A self-decoder in the bottom half produces one global KV cache; a cross-decoder in the top half never produces keys or values at all and attends to that single cache through cross-attention. The cache is now \(O(1)\) in depth rather than \(O(L)\), and because the upper half has nothing to prefill, prefill can exit early without changing the output (Sun et al., 2024, You Only Cache Once, arXiv:2405.05254).

The distinction that matters operationally: CLA saves memory, YOCO saves memory and prefill compute. Gemma 3n took the YOCO-shaped version, sharing the keys and values of a middle layer with all the layers above it, and reports 2x faster prefill than Gemma 3 4B, which is the number that shows up as time-to-first-token on a phone (Google, 2025, Introducing Gemma 3n: the developer guide).

Where the KV layers sit

The obvious question is which layers get to own a cache. A systematic study names three families: pizza puts the KV-producing layers at the bottom, sandwich at both ends, lasagna spreads them uniformly. At a 2x reduction most layouts beat a standard transformer on throughput while holding downstream accuracy. Push further and the layouts separate: pairing the queries of every layer with keys and values from the upper layers holds accuracy best, at the price of extra training cost and slower prefill, because an upper-layer cache cannot be filled until the tokens have been pushed that far up (Wu, Wu and Tu, 2024, A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference, arXiv:2410.14442).

The techniques stack, which is the real argument for them. Character.AI's production serving stack combines MQA on every attention layer, a 1-in-6 ratio of global to local attention, and cross-layer KV sharing among the global layers, and reports a cumulative KV cache reduction of more than 20x with no quality regression (Character.AI, 2024, Optimizing AI Inference at Character.AI).

When it breaks

You cannot retrofit it. Tying K and V across layers changes what each layer's attention can express, so it is a pretraining decision, not a serving flag. Post-hoc variants exist, but they require continued pretraining to recover quality, which puts them in the same cost bracket as the uptraining recipe GQA needs.

Prefill and decode pull in opposite directions. A layout that gives the upper layers their own caches prefills fast and stores more; a layout that forces every query to read an upper-layer cache stores little and prefills slowly. Which one wins depends on your prefill-to-decode token ratio, and a RAG workload with 20k-token prompts and 200-token answers sits at the opposite end of that spectrum from a chat workload.

Shared caches break per-layer cache tricks. Eviction, quantisation and offload policies that assume each layer's cache is independent (see KV cache eviction and compression) become coupled: evicting a token from a shared cache evicts it from every layer that reads it. Aggressive sharing plus aggressive eviction is where quality regressions appear that neither change causes alone.

The savings are bounded by what else you already did. If attention is already MQA with a 1024-token sliding window on five of every six layers, the cache is small and depth-sharing has little left to take. Measure the current cache breakdown before designing for this; the layers you would share are often not the ones holding the bytes.