Context Windows and Long-Context Models

The advertised context length is a marketing number and an engineering ceiling; the usable context is something smaller and harder to measure. A model can accept a million tokens, keep its perplexity stable across all of them, and still fail to answer a question whose evidence sat in the middle of the input. Two failure surfaces hide behind one number. The first is whether the model can represent and process a sequence that long at all. The second is whether it actually attends to the right part of it. Confusing the two is the most common mistake teams make when they buy "long context" and wonder why retrieval still beats it.

What sets the ceiling

Three pressures bound how long a window you can train and serve.

1. Attention compute is O(n^2). Every token attends to every prior token, so doubling the sequence quadruples the attention cost (see attention-mechanism). At 100k+ tokens this dominates the forward pass, which is why long context and inference-optimisation work are joined at the hip.
2. KV-cache memory is O(n). During generation the model caches a key and value vector per layer per token. That cache grows linearly and, at long context, dwarfs the model weights in memory; it is the real reason a 1M-token request is expensive to serve (see kv-cache).
3. Positional representability. The encoding has to stay sane at positions the model never trained on, which is the entire job of RoPE interpolation and YaRN (see positional-encodings).

Clear all three and you have a window that is long on paper. None of them guarantees the model uses it well.

Lost in the middle

The sharpest evidence for the gap is "Lost in the Middle". Across multi-document QA and key-value retrieval, models recall information best when it sits at the start or end of the input and measurably worse when the same fact sits in the middle, tracing a U-shaped accuracy curve. The effect persists in models explicitly built and advertised for long context. So a document stuffed with 80k tokens of preamble before the relevant clause is a setup for failure, not because the model cannot hold 80k tokens, but because it systematically under-weights the centre. This single result is why retrieval-augmented generation (see retrieval-augmented-generation) keeps winning: pulling the right 2k tokens to the top of the prompt sidesteps the positional bias entirely.

Attention sinks

Part of why long context is fragile shows up in a strange empirical fact: transformers dump a large, near-constant share of attention onto the first few tokens, regardless of whether those tokens carry any meaning. StreamingLLM named these attention sinks and explained the mechanism. Softmax forces the attention weights to sum to one, so when a head has nothing it genuinely wants to look at, it has to put that mass somewhere; the initial tokens, always visible under the causal mask, become the default dump. The practical consequence is concrete: naive sliding-window attention that evicts those first tokens to save memory causes quality to fall off a cliff, because you threw away the sink the model relies on. Keep a handful of initial tokens plus a recent window and a model trained on finite context streams over millions of tokens without fine-tuning.

Extending the window

Two distinct strategies push the ceiling up:

• Re-fit the positions. Position interpolation and YaRN remap long positions back into the trained range with a short fine-tune, the standard way a short-context base becomes a long-context release.
• Distribute the sequence. Ring Attention shards a long sequence across devices and overlaps the communication of key-value blocks with blockwise attention compute, so the effective window scales with the device count without approximating the attention itself. This is how training and serving reach multi-million-token contexts.

Architectural bets (sub-quadratic and recurrent designs like Mamba) attack the O(n^2) wall directly, but the transformer-plus-extension path above is what ships in most frontier long-context models today.

When it falls down

• Advertised length is not effective length. "Needle in a haystack" tests, where one planted fact is retrieved, flatter a model; real tasks need it to integrate many facts across the window, where the lost-in-the-middle penalty bites. Treat the published number as a ceiling, not a capability.
• Cost scales the wrong way. A near-full window pays the O(n^2) attention tax and a large KV cache on every token generated. Long context is frequently the most expensive way to solve a problem that a retrieval step would solve more cheaply and more accurately.
• Position in the prompt is a lever. Because of the U-shaped curve, where you place the critical instruction or evidence (top or bottom, not buried) materially changes the answer. This is a prompt-engineering decision, not just an infrastructure one.
• Extended windows can dilute. A context stretched with interpolation or YaRN sometimes reasons more weakly per token than the base model did, trading depth of use for breadth of reach.

Further reading

• Lost in the Middle: How Language Models Use Long Contexts - the U-shaped recall curve and the multi-document QA experiments.
• Efficient Streaming Language Models with Attention Sinks - StreamingLLM; the attention-sink phenomenon and why evicting initial tokens breaks window attention.
• Ring Attention with Blockwise Transformers for Near-Infinite Context - distributing a single long sequence across devices without approximating attention.
• YaRN: Efficient Context Window Extension of Large Language Models - extending a trained RoPE window with minimal fine-tuning.