Context Rot

The naive mental model of a context window is a bucket: it holds N tokens and trouble starts only above N. The measured reality is worse. As the number of tokens in the window grows, a model's ability to accurately use that context decreases, well before the hard limit, and it decreases unevenly. This phenomenon has a name now, context rot, and it is the reason "just use a bigger window" is not the answer it appears to be.

What the measurements show

A controlled study across 18 models (including GPT-4.1, Claude 4, and Gemini 2.5) found that accuracy on even simple retrieval tasks falls as input length grows, and crucially that it does not fall on a gentle linear slope (Chroma, 2025, Context Rot). Models hit cliffs: some are fine at 32k tokens and collapse at 64k; others hold and then drop off suddenly. Accuracy losses of roughly 20 to 50% between 10k and 100k+ tokens were common on needle-in-a-haystack style tasks, and the degradation was sensitive to how similar the distractors were to the target and how the surrounding "haystack" was structured.

This complements the older lost-in-the-middle result: recall is not uniform across positions, and it is not uniform across lengths either.

Why it happens

Two contributing causes. First, attention creates \(n^2\) pairwise relationships across \(n\) tokens, so a longer sequence spreads the model's focus thinner over more competing candidates. Second, models are trained predominantly on shorter sequences, so they have less experience with, and fewer specialized parameters for, genuinely long-range dependencies. The window can hold the tokens; the model's attention was never well-calibrated to use all of them at once.

A bigger window is a bigger desk, not a better reader

The practical reframe: you are not trying to stay under a hard cap, you are trying to keep the window in the region where the model still reasons well, which is often a fraction of the advertised length. Tokens are not free even when they fit. This is the empirical bedrock under context engineering: if extra tokens can actively hurt, then aggressively evicting stale ones is not premature optimization, it is correctness.

The multi-turn version

Context rot is usually framed as a long-input problem, but a sharper cousin appears in multi-turn use. Splitting a single well-specified request into sequential turns caused an average 39% quality drop across six tasks, because the model's early, incomplete attempts stayed in the context and dragged down the final answer (Laban et al., 2025, LLMs Get Lost in Multi-Turn Conversation, arXiv:2505.06120). The window was not even long here; the damage came from what accumulated, not how much. Stale wrong attempts are toxic context.

Why benchmarks hide it

Needle-in-a-haystack tests, where one planted fact must be retrieved from filler, systematically flatter models, because real tasks require integrating many facts across a long, self-generated, sometimes-contradictory history. A model that aces a synthetic haystack can still fall over on a real long run. Treat the advertised context length as a ceiling on what fits, never as a promise of what the model will actually use.

Further reading

• Chroma Research, 2025, Context Rot: How Increasing Input Tokens Impacts LLM Performance - the controlled measurement across 18 models.
• Laban et al., 2025, LLMs Get Lost in Multi-Turn Conversation, arXiv:2505.06120 - the 39% multi-turn degradation and its causes.
• Liu et al., 2023, Lost in the Middle, arXiv:2307.03172 - the positional precursor to context rot.