Attention Sinks

Look at the attention weights of a trained transformer on almost any input, and one pattern shows up over and over: the very first few tokens of the sequence receive a large share of attention from nearly every later position, regardless of what those first tokens actually are. This holds even when the first tokens are semantically empty, a beginning-of-sequence marker, or unrelated to anything downstream. Xiao et al., 2023 named this phenomenon attention sinks and turned an odd observation into a practical fix for a real deployment problem: language models that fall apart when you try to stream-generate past their training context length with a naive sliding cache.

Why softmax needs somewhere to put the leftover weight

Softmax attention weights must sum to exactly 1 at every position (see softmax-logits); there is no option to assign low total attention when nothing is particularly relevant. If a query position genuinely has no strong match among the available keys, the softmax still has to distribute its full probability mass somewhere. Xiao et al.'s explanation is that the model learns to treat certain positions, empirically almost always the earliest ones, as a low-cost dumping ground for this excess attention mass: attending heavily to a token whose value vector contributes little useful content is a way to keep the output roughly unchanged while still satisfying the sum-to-one constraint. Early tokens make convenient sinks because, being visible to every later position under causal masking, they are the one thing every query can always reliably attend back to.

Why naive streaming breaks without them

A natural way to let an LLM generate indefinitely with a fixed memory budget is a sliding KV cache: keep only the most recent N tokens' keys and values, evict older ones as new tokens arrive (see kv-cache and sliding-window-attention). Xiao et al. found this collapses model quality catastrophically once the earliest tokens, the attention sinks, get evicted from the window. It is not that the content of those early tokens is missed; it is that the model has nowhere left to route the excess softmax mass, and the whole attention distribution destabilises across every later layer.

StreamingLLM: keep the sink, slide the rest

The fix in Xiao et al., 2023 (StreamingLLM) is almost embarrassingly simple once the mechanism is understood: permanently retain the keys and values of the first few tokens (four is often enough) in the cache, no matter how long generation runs, and slide the eviction window only over everything after them.

cache = [sink_tokens (fixed, first ~4)] + [sliding_window (most recent N)]

With the sink tokens preserved, models that were only ever trained on sequences up to length L generate stably for millions of tokens past L, at constant memory, with no retraining and no fine-tuning required. This is a striking result: a phenomenon that looks like a training artifact turns out to be load-bearing infrastructure the model depends on, and once you know to protect it explicitly, a purely inference-time cache policy recovers stability that a naive sliding window destroys.

A caveat the paper is explicit about

StreamingLLM restores fluent, stable generation at long lengths; it does not give the model access to the content of evicted tokens. A question whose answer depended on a fact that fell out of the sliding window will not be answerable, because that information is genuinely gone from the cache. Attention sinks fix a stability problem, not a memory problem; true long-range recall over evicted content still requires an architecture change (larger context, retrieval, or a compression mechanism), not just sink preservation.

When it falls down

• Sinks stabilise, they do not remember. As above: content evicted from the sliding window is unavailable to later generation, no matter how well the sink tokens are preserved. Do not read StreamingLLM as a substitute for genuine long-context modelling.
• Which tokens become sinks is not fully predictable. The first few positions dominate empirically and reliably enough to be exploited, but the underlying mechanism (why training settles on early positions specifically, rather than some other stable dumping ground) is an empirical finding, not something derived from first principles.
• Sink behaviour is a property of trained models, not the architecture in the abstract. Models trained differently, or with explicit sink tokens designed in from the start (some follow-up work adds a dedicated learned sink token), can behave differently; treating "first four tokens" as a universal constant across all transformers is an approximation, not a law.
• It interacts with other cache-compression tricks. Combining attention-sink preservation with quantised or grouped-query KV caches (see multi-query-grouped-query-attention) generally works but adds engineering complexity: the sink tokens still need protecting from whatever eviction or compression policy governs the rest of the cache.

Further reading

• Xiao et al., 2023, Efficient Streaming Language Models with Attention Sinks, arXiv:2309.17453 - names and explains attention sinks, and introduces StreamingLLM.