Length Extrapolation

Length extrapolation, precisely defined, means running a model on sequences longer than anything in its training data, with no additional fine-tuning, and asking whether output quality degrades gracefully, catastrophically, or barely at all. It is a narrower question than "can this model handle long context" in general, because it isolates zero-shot generalisation from the deliberate, fine-tuned extension techniques covered in RoPE scaling: NTK-aware and YaRN. Extrapolation asks what the same weights do, unmodified, outside their training distribution; scaling methods ask how cheaply you can adapt those weights to a new target length.

How the major schemes fare with zero adaptation

• Learned absolute positions fail hard. There is no table row past the trained maximum, so there is no valid computation to run at all, not degraded output, no output.
• Sinusoidal encoding is defined at any position but degrades quickly, because the trained attention and feed-forward weights never saw the frequency combinations that appear far past training length (see sinusoidal positional encodings).
• RoPE degrades more gracefully over a moderate range but still suffers, specifically from high-frequency dimension pairs aliasing into ambiguous angles once you exceed training length.
• ALiBi extrapolates well on perplexity by design; its linear bias is well-defined and monotonic at any distance, with nothing to alias (see ALiBi).
• NoPE shows the strongest zero-shot length generalisation on synthetic algorithmic tasks in the literature that introduced it, apparently because the causal mask alone supplies enough implicit position signal (see the NoPE result).

The metric trap: flat perplexity is not the same as useful context

A model can maintain low perplexity at 16000 tokens while effectively ignoring most of that context, relying mostly on a local window and general language statistics to predict the next token. Perplexity, averaged over an entire long sequence, cannot distinguish "used the whole context well" from "used the last few hundred tokens well and got lucky that most next-token predictions don't require anything further back". This is a real and common failure mode; a scheme can win on perplexity-based extrapolation claims while still failing badly on tasks that require genuine long-range retrieval.

Needle-in-a-haystack evaluation

The standard corrective is task-based: insert one specific, checkable fact at a controlled depth inside an otherwise irrelevant long context, vary both total context length and insertion depth, and measure whether the model can retrieve the fact. This separates "the model doesn't crash or degrade numerically" (extrapolation in the stability sense) from "the model actually retrieves distant information" (extrapolation in the sense that matters for real tasks). Results across many models and positional schemes show a recurring pattern, retrieval accuracy is often worse for facts placed in the middle of a long context than for facts at the very start or very end, independent of which positional scheme is in use (see effective vs nominal context length for the broader framing).

Why extrapolation is hard in general, not just an architecture choice

Extrapolation is fundamentally an out-of-distribution generalisation problem. Even a scheme with an exact algebraic relative-distance guarantee, like RoPE, was only ever optimised against distance values up to the training length; nothing about the guarantee ensures the trained attention weights respond sensibly to distance values several times larger, even though the encoding itself remains numerically well-defined at any distance. The positional scheme controls what information is representable; training data and objective control what the network actually learns to do with it.

When it falls down

• "Extrapolates on perplexity" and "extrapolates on retrieval" are different claims. Papers and model cards sometimes report only the easier one, and it is worth checking which.
• Extrapolation is data-and-training dependent, not purely architectural. Two models with identical positional schemes but different pretraining mixes can extrapolate very differently.
• Synthetic benchmarks do not always predict natural-language performance. A scheme that wins on algorithmic copy or lookup tasks can lag on long natural-document comprehension, and vice versa.
• Reported zero-shot extrapolation is usually modest in scale. Most claims cover 2x to 4x training length; almost none hold up unmodified at 100x, which is why production long-context releases combine extrapolation-friendly schemes with explicit fine-tuned scaling rather than relying on extrapolation alone.

Further reading

• Press et al., 2021, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation - the paper that made train-short-test-long a named benchmark.
• Kazemnejad et al., 2023, The Impact of Positional Encoding on Length Generalization in Transformers - systematic comparison of extrapolation across positional schemes on algorithmic tasks.