The Wavelength Problem: How One Number Decides a Model's Context Length

Llama 2 shipped with rope_theta = 10000.0. Llama 3 shipped with rope_theta = 500000.0. The attention code around it is nearly identical. That single constant is a large part of the difference between a model that loses a fact 6,000 tokens back and one that retrieves it at 32,000 (Grattafiori et al., 2024, The Llama 3 Herd of Models, arXiv:2407.21783).

Most explanations of long context reach for architecture: sparse attention, sliding windows, retrieval augmentation, state-space hybrids. Those are real, and they are mostly about cost. The thing that decides whether a transformer can tell two distant positions apart at all is smaller and less discussed: the set of frequencies that rotary position embedding assigns to each coordinate pair of each attention head. Change the base and you change every one of them. Almost every long-context technique published since 2023 is a specific edit to that spectrum, and the ones that work are the ones that edit it non-uniformly.

Why this matters: Position is not stored in a transformer, it is encoded as a frequency. Once you can read a model's RoPE spectrum you can predict, before running a single evaluation, roughly where its retrieval will fail, why perplexity will not tell you, and which context-extension method will and will not survive the stretch.

TL;DR

• RoPE assigns coordinate pair \(i\) a wavelength \(\lambda_i = 2\pi b^{2i/d}\). For a 128-dim head at \(b = 10^4\), that spans 6.3 tokens to about 54,400. The base fans the spectrum out from a fixed high-frequency anchor; it does not shift it.
• Pairs whose wavelength exceeds the training length never complete a rotation during training. At \(b=10^4\), \(d=128\), \(L=4096\), that is 18 of 64 pairs, and those 18 are the ones that go out of distribution when you extrapolate.
• There is a lower bound on the base, not just an upper one. Men et al. derive numerically that 32k context requires a base of at least \(6.4 \times 10^5\) and 128k at least \(7.8 \times 10^6\) (arXiv:2405.14591).
• Below the bound, the failure is invisible in perplexity. A model fine-tuned at 32k with an insufficient base held a healthy loss and could not retrieve a planted fact at a distance of 1,000 tokens.
• Llama 3's 500,000 lands just under the derived 32k threshold, and Llama 3's own paper says it was validated for lengths up to 32,768. Two independent methods converge on the same number.
• Position Interpolation, NTK-aware scaling, and YaRN are three different answers to one question: which frequencies do you sacrifice. PI sacrifices all of them uniformly, NTK sacrifices the slow end, YaRN sacrifices the slow end and leaves the fast end untouched by construction.
• RoPE does not work by decaying attention with distance. Barbero et al. prove the expected decay is zero for Gaussian queries and keys, and find that models use the highest frequencies for sharp positional heads and the lowest for distance-agnostic semantics (arXiv:2410.06205).
• The 2025 frontier answer is to stop applying one spectrum everywhere: Llama 4 and Cohere's Command A both interleave windowed RoPE layers with full-attention layers that carry no positional encoding at all.

At a Glance

flowchart LR
  T["Token at position m"] --> S["Split head into d/2 pairs"]
  S --> F["Pair i rotates by m·b^(-2i/d)"]
  F --> H["High freq pairs<br/>wavelength ~6 tokens"]
  F --> L["Low freq pairs<br/>wavelength ~54k tokens"]
  H --> A["Sharp local attention<br/>diagonal, prev-token heads"]
  L --> D["Distance-agnostic<br/>semantic channels"]
  A --> Q["Attention logit"]
  D --> Q
  B["base b"] -.->|"sets the whole spectrum"| F
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class T,S blue
  class F,H,L purple
  class A,D,Q teal
  class B amber

The base is the only knob on the left-hand side of this picture, and everything to the right of it is downstream.

[IMAGE: Log-scale horizontal axis from 1 to 10,000,000 tokens. Two stacked strips of 64 tick marks each, one for base 10,000 and one for base 500,000, showing each coordinate pair's wavelength. A vertical line marks the 4,096-token training length. Caption: "Raising the base fans the spectrum rightwards from a fixed anchor at 2π tokens; the ticks that cross the training-length line are the pairs that never complete a rotation."]

Before Rotation: Twenty Years of Telling a Transformer Where It Is

Self-attention is a set operation. Permute the input and the output permutes with it, exactly, which means a transformer with no positional mechanism cannot distinguish "dog bites man" from "man bites dog". Every scheme in the field's history is an answer to that, and the answers keep swinging between two poles: put position in the representation, or put it in the attention score.

timeline
    title From additive signals to frequency engineering
    2017 : Vaswani et al. add fixed sinusoids to the input embedding
         : Position lives in the representation and is hoped to survive the layers
    2018 : Shaw et al. inject learned relative offsets directly into attention scores
         : BERT ships a learned absolute position table with a hard length cap
    2021 : Su et al. publish RoFormer, rotating query and key by angle proportional to position
         : Press et al. publish ALiBi, a zero-parameter linear distance penalty per head
    2023 : Position Interpolation squeezes long positions into the trained angular range
         : NTK-aware scaling and then YaRN interpolate the spectrum non-uniformly
         : Code Llama simply sets the base to 1,000,000 and fine-tunes at 16k
    2024 : Llama 3 ships base 500,000 with six-stage context extension to 128k
         : Men et al. derive a lower bound on the base; Barbero et al. dismantle the decay story
    2025 : Llama 4 and Command A interleave RoPE layers with no-position layers
         : Positional encoding stops being one global choice and becomes a per-layer one

Sinusoidal encodings (Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762) put a fixed pattern of sines and cosines into the input embedding and hoped the network would extract relative offsets from it. It works, in the sense that the model learns something, and it is fragile: the signal has to survive twelve or ninety layers of mixing while competing with semantic content for the same coordinates.

Shaw et al. moved position into the score, adding a learned bias that depends on \(i - j\) (Shaw et al., 2018, arXiv:1803.02155). That is closer to what position means for attention, and it costs an extra term in the innermost loop.

RoPE's contribution was to get relative behaviour out of an absolute operation (Su et al., 2021, RoFormer, arXiv:2104.09864). Rotate \(\mathbf{q}\) at position \(m\) and \(\mathbf{k}\) at position \(n\) by their own angles; the dot product of two rotated vectors depends only on \(m - n\). Absolute rotation in, relative dependence out, no extra term in the score, no table to run out of. The paper's own justification leaned on a long-term decay property, which turned out to be the wrong explanation for the right method, a point the field only clarified in 2024.

How the Base Frequency Actually Works

The spectrum, read as wavelengths

RoPE splits each head's \(d\) dimensions into \(d/2\) coordinate pairs and rotates pair \(i\) by \(m\theta_i\) at position \(m\), where

The dot product between a query at \(m\) and a key at \(n\) becomes a sum over pairs of terms in \(\cos((m-n)\theta_i)\), so each pair contributes a sinusoid in relative distance. The natural unit is the wavelength: the distance over which pair \(i\) completes one full turn.

Two consequences follow immediately, and both are usually missed.

First, \(\lambda_0 = 2\pi \approx 6.3\) tokens regardless of the base, because \(b^0 = 1\). The base does not translate the spectrum; it anchors one end and stretches the other. Raising \(b\) from \(10^4\) to \(5 \times 10^5\) leaves the finest positional resolution exactly where it was and pushes the coarsest from about 54,400 tokens to about 2.56 million.

[IMAGE: A unit circle for each of four coordinate pairs (i = 0, 16, 32, 63), showing where the query vector points at positions m = 0, 100, 1000, and 4000. Pair 0's marker has wrapped hundreds of times and lands anywhere; pair 63's marker has barely left the starting angle. Caption: "The same position index produces a completely different amount of rotation in each pair. Only the pairs that wrap have seen the whole circle."]

Second, the spectrum is geometric, not linear. Pairs are spaced by a constant ratio, so the model has fine resolution densely packed at short distances and progressively coarser resolution at long ones. That is a sensible prior for language and it is why the encoding degrades gracefully rather than catastrophically as distance grows.

The critical dimension

During pretraining at length \(L\), a pair with \(\lambda_i < L\) sweeps through complete periods and observes every angle on the circle. A pair with \(\lambda_i > L\) observes only one short arc; the angle \(m\theta_i\) never exceeds \(2\pi L / \lambda_i < 2\pi\).

At inference beyond \(L\), exactly those slow pairs are queried at angles no training example ever produced. Liu et al. name the boundary the critical dimension and build a quantitative account of extrapolation around it (Liu et al., 2023, Scaling Laws of RoPE-based Extrapolation, arXiv:2310.05209). Their finding is genuinely counterintuitive: fine-tuning with either a smaller or a larger base improves extrapolation, and for opposite reasons. A small base pulls every pair inside the training window, so nothing is ever out of distribution, at the price of long-distance discrimination. A large base delays the wrap-around, so distinct long distances stay distinguishable, at the price of needing training at the target length.

Between those two regimes lies a middle where a model is neither fully trained on its slow pairs nor safely below the wrap-around, and that middle is where most naively-extended models sit.

The bound nobody expected

If small bases avoid out-of-distribution angles, why did the field move upward? Because a second constraint binds harder, and it binds from below.

Men et al. ask what it takes for attention to prefer a semantically similar token over a random one at distance \(m\). In expectation, that reduces to the sign of

While \(B_{m,\theta} \ge 0\) the similar token wins. Once it goes negative, random tokens at that distance score higher than similar ones, and no amount of training repairs a geometry that is actively inverted. Solving numerically for the smallest base that keeps \(B_{m,\theta}\) non-negative across a target length gives an absolute floor (Men et al., 2024, Base of RoPE Bounds Context Length, arXiv:2405.14591).

The reason this result mattered is not the theorem, it is the experiment attached to it. A model fine-tuned at 32k with a base below the bound kept a perfectly healthy perplexity and could not retrieve related information at a context length as short as 1k. Perplexity is dominated by the nearest few hundred tokens, and the fast pairs that serve those tokens are unaffected by the base. So the metric that everyone plots stays flat while the capability everyone claims quietly disappears.

[IMAGE: Line plot of \(B_{m,\theta} = \sum_i \cos(m\theta_i)\) against relative distance \(m\) on a log x-axis from 1 to 200,000, drawn for three bases (10⁴, 5×10⁵, 10⁷) at d=128. A horizontal zero line is emphasised, and the crossing point of each curve is annotated. Caption: "The distance at which the curve goes negative is the distance past which random tokens outscore similar ones. Raising the base pushes the crossing to the right."]

What interpolation actually does to the spectrum

Once you accept that the base sets a spectrum, the three main extension methods stop looking like alternatives and start looking like three choices about which frequencies to sacrifice.

flowchart TB
  subgraph PI["Position Interpolation"]
    P1["Divide every position by s"] --> P2["All wavelengths x s"]
    P2 --> P3["Fine resolution crushed<br/>6.3 tokens becomes 50"]
  end
  subgraph NTK["NTK-aware base scaling"]
    N1["Raise base to b·s^(d/(d-2))"] --> N2["Fast pairs unchanged"]
    N2 --> N3["Slow pairs stretched by s"]
  end
  subgraph YARN["YaRN"]
    Y1["Count rotations per pair"] --> Y2["r above 32: leave alone"]
    Y1 --> Y3["r below 1: interpolate fully"]
    Y1 --> Y4["between: ramp"]
    Y4 --> Y5["Plus attention temperature"]
  end
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class P1,P2,P3 rose
  class N1,N2,N3 amber
  class Y1,Y2,Y3,Y4,Y5 emerald

Position Interpolation divides the position index by a scale \(s\) before rotating, so position 30,000 is presented as 3,750 and every angle stays inside the trained range. Chen et al. prove the interpolated attention-score bound is at least about 600 times smaller than the extrapolated one, and extend LLaMA to 32,768 tokens within 1,000 fine-tuning steps (Chen et al., 2023, arXiv:2306.15595). The cost is uniform: every wavelength stretches by \(s\), including the 6.3-token pair that was doing the fine-grained local work.

NTK-aware scaling started as a community post and became standard practice: instead of dividing positions, raise the base to \(b \cdot s^{d/(d-2)}\). Because the spectrum is anchored at \(\lambda_0 = 2\pi\), this leaves the fastest pair untouched and stretches the slowest by exactly \(s\), spreading the compression across the spectrum in proportion to how slow each pair already was. Code Llama took the blunt version of the same idea, setting the base directly to \(10^6\) and fine-tuning at 16k (Rozière et al., 2023, arXiv:2308.12950).

YaRN makes the choice explicit rather than implicit. It computes, for each pair, the number of rotations \(r(i) = L / \lambda_i\) that pair completes over the original context, then applies a ramp: pairs with \(r > \beta = 32\) are left alone entirely, pairs with \(r < \alpha = 1\) are interpolated by the full factor \(s\), and pairs in between are blended. It also scales attention logits by a temperature \(\sqrt{1/t} = 0.1 \ln(s) + 1\) to counteract the entropy increase from softmaxing over more keys. The result extends Llama 2 7B and 13B to 64k and then 128k using, by the authors' accounting, 10x fewer tokens and 2.5x fewer training steps than prior methods, at 400 to 600 A100-hours (Peng et al., 2023, YaRN, arXiv:2309.00071).

The decay story was wrong

RoFormer justified RoPE partly by a long-term decay property: attention between distant tokens should weaken naturally. It is an appealing story and the field repeated it for three years.

Barbero et al. show it is not what happens. With queries and keys drawn from a Gaussian, the expected value of the RoPE-modulated activation shows zero decay with distance, so decay cannot be the mechanism (Barbero et al., 2024, Round and Round We Go, arXiv:2410.06205). What they find instead is a division of labour across the spectrum. The highest frequencies are what allow a head to build an arbitrarily sharp attention pattern, and they prove that diagonal and previous-token heads are constructible with high-frequency RoPE and provably not constructible without positional information. The lowest frequencies, meanwhile, are preferred by the trained model for carrying semantic content that should be distance-agnostic.

They test the conclusion by removing the lowest frequencies entirely, a scheme they call p-RoPE. Training Gemma 2B from scratch, 0.75-RoPE matched or slightly beat standard RoPE with \(\theta = 500\text{k}\): validation perplexity 4.4414 against 4.4485 on Wiki, and 6.4422 against 6.4593 on FlanV2. Small margins, and the direction is the point: the slowest quarter of the spectrum was not carrying its weight.

That reframes every base-scaling decision. Raising the base is not "adding long-range capability", it is reallocating pairs from the fast end's job to the slow end's job. If the slow end is largely serving as distance-agnostic semantic bandwidth, there is a limit to how much of the head you want to spend on it.

Seeing It in Motion

Extending a model's context is not one action but a loop, and the loop is where the practical failures live.

stateDiagram-v2
    [*] --> Base: pretrained at 8k
    Base --> Rescale: pick target length
    Rescale --> Tune: apply PI / NTK / YaRN
    Tune --> ShortEval: brief fine-tune at target
    ShortEval --> Retrieval: short-context scores recovered?
    ShortEval --> Tune: regression, more tuning
    Retrieval --> Advance: needle solved at this length?
    Retrieval --> Rescale: fails, base too small
    Advance --> Rescale: next stage
    Advance --> [*]: target reached

That is close to what Llama 3 actually did. Context rose in six stages from 8K to 128K over roughly 800B tokens, and a stage advanced only when short-context evaluations had fully recovered and needle-in-a-haystack was solved at that length. Two gates, both necessary, because either one alone passes models that are broken in the other direction.

The per-layer view has changed too. Until 2025 the spectrum was a global property of a checkpoint. It is now a per-layer choice.

graph TD
  IN["Token stream"] --> L1["Layer 1: RoPE + 4k window"]
  L1 --> L2["Layer 2: RoPE + 4k window"]
  L2 --> L3["Layer 3: RoPE + 4k window"]
  L3 --> L4["Layer 4: NoPE + full attention"]
  L4 --> REP["Repeat block"]
  REP --> OUT["Logits"]
  KV1["Bounded KV: 4k per layer"] -.-> L1
  KV1 -.-> L2
  KV1 -.-> L3
  KV2["Unbounded KV: grows with context"] -.-> L4
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class L1,L2,L3 purple
  class L4 teal
  class KV1 emerald
  class KV2 amber

Cohere's Command A uses exactly this 3:1 arrangement, sliding-window RoPE on three layers out of four and full-attention NoPE on the fourth, with a 4,096-token window (Cohere, 2025, Command A Technical Report). Meta's iRoPE in Llama 4 is the same shape, placing NoPE layers every fourth layer and crediting the arrangement for Scout's 10-million-token window (Meta, 2025, The Llama 4 herd). The accompanying study from Cohere sweeps the design space and reports the hybrid beating full-attention RoPE baselines on both long and short tasks (Yang et al., 2025, Rope to Nope and Back Again, arXiv:2501.18795).

The reason removing the position signal helps is that a NoPE layer has nothing to extrapolate. Causal masking already breaks permutation symmetry, since token \(m\) attends over \(m\) predecessors, and Kazemnejad et al. prove that a decoder-only transformer without positional encoding can represent both absolute and relative schemes, and measure it generalising to unseen lengths better than APE, ALiBi, or RoPE on downstream tasks (Kazemnejad et al., 2023, NeurIPS, arXiv:2305.19466).

[IMAGE: Side-by-side attention heatmaps for the same 16k prompt. Left: a windowed RoPE layer, showing a bright band hugging the diagonal and nothing beyond the 4k window. Right: a NoPE full-attention layer, showing diffuse mass with a distinct bright column at the position of a planted fact. Caption: "The two layer types are not redundant; each covers the other's blind spot."]

By the Numbers

The derived floor on the base, alongside what shipped models actually chose:

Read the second row carefully: Llama 2's base of \(10^4\) sits below the derived floor for its own 4,096-token context window, which is consistent with the well-documented observation that Llama 2 degrades well before its nominal limit. Read the last two rows carefully too. Llama 3.1 ships 128k on a base that the analysis says supports 32k, and it works, because staged pretraining is doing work that the static bound does not model. The bound describes what the geometry permits, not what training extracts from it.

Wavelengths for a 128-dimensional head, which is where the abstraction becomes concrete:

Sources: derived bounds from Men et al. (arXiv:2405.14591, Table 2). Model bases from the Llama 3 paper (arXiv:2407.21783), RoFormer (arXiv:2104.09864), and the published Llama 2 configuration. Wavelengths computed directly from \(\lambda_i = 2\pi b^{2i/d}\) at \(d=128\) and rounded to three significant figures. The alignment between the derived 32k floor and Llama 3's validated 32,768 is an observation about two independent results, not a claim that either caused the other.

[IMAGE: Table-style figure or grouped bar chart plotting derived minimum base (log y-axis) against target context length (log x-axis), with points overlaid for Llama 2, Code Llama, Llama 3, and Llama 3.1 marked by their actual base and actual claimed context. Caption: "Every shipped model sits somewhere relative to the line; the ones below it lean on staged training to make up the difference."]

A Concrete Example

Take a 128-dimensional head from a model pretrained at 4,096 tokens with base 10,000, and extend it to 32,768. Scale factor \(s = 8\).

Step 1 — read the spectrum. Since \(10000^{-i/64} = 10^{-i/16}\), the frequencies are clean powers of ten. Pair 0 has \(\theta_0 = 1\) and \(\lambda_0 = 6.28\) tokens. Pair 32 has \(\theta_{32} = 0.01\) and \(\lambda_{32} = 628\) tokens. Pair 63 has \(\theta_{63} = 1.15 \times 10^{-4}\) and \(\lambda_{63} = 54{,}410\) tokens.

Step 2 — find the critical dimension. Solve \(2\pi \cdot 10^{i/16} \le 4096\). That gives \(10^{i/16} \le 651.9\), so \(i/16 \le 2.814\) and \(i \le 45.03\). Pairs 0 through 45, forty-six of them, complete at least one full rotation during pretraining. Pairs 46 through 63, eighteen of them, never do. Those eighteen are the entire out-of-distribution surface.

Step 3 — Position Interpolation. Divide positions by 8. Token 30,000 is presented at effective position 3,750, comfortably in range, and every wavelength effectively multiplies by 8. Pair 0 now completes a rotation over 50.3 tokens instead of 6.28. The angular separation between adjacent tokens in the fastest pair drops from 1 radian to 0.125, an eightfold loss of local resolution in exactly the pairs Barbero et al. identify as building sharp local attention. The model gets its long context and reads adjacency more coarsely than it was trained to.

Step 4 — NTK-aware scaling instead. Set \(b' = b \cdot s^{d/(d-2)} = 10000 \times 8^{128/126} = 10000 \times 8.268 = 82{,}680\). Check both ends. Pair 0: \(\theta'_0 = 82680^0 = 1\), unchanged, \(\lambda_0\) still 6.28 tokens. Pair 63: \(\theta'_{63} = 82680^{-63/64} = 1.443 \times 10^{-5}\), so \(\lambda_{63} = 435{,}500\) tokens, precisely \(8\times\) the original. The compression has been moved entirely off the fast end and onto the slow end, which is exactly the design intent.

Step 5 — the check almost nobody runs. The new base is 82,680. The derived floor for a 32k context is \(6.4 \times 10^5\), roughly eight times higher. The comparison is not perfectly apples to apples, since the bound is derived for a model trained at that length with that base rather than for an NTK-rescaled one, but the gap is large and in the direction that predicts trouble. It is a plausible mechanical account of why NTK-scaled models routinely pass perplexity and needle tests at 32k and then underperform on multi-hop retrieval at the same length.

Step 6 — YaRN's partition. Compute rotations per pair over the original context, \(r(i) = 4096 / \lambda_i\). Pair 0: \(r = 651.9\), far above \(\beta = 32\), so \(\gamma = 1\) and it is left completely alone. Pair 63: \(r = 0.0753\), below \(\alpha = 1\), so \(\gamma = 0\) and it is interpolated by the full factor 8. The boundaries fall at \(\lambda = 4096/32 = 128\) tokens, which is \(i \approx 20.9\), and at \(\lambda = 4096\), which is \(i \approx 45.0\). So YaRN splits the 64 pairs into three groups: pairs 0 to 20 untouched, pairs 21 to 45 ramped, pairs 46 to 63 fully interpolated. The fully-interpolated set is exactly the eighteen pairs from step 2 that never completed a rotation. YaRN is not approximating the critical dimension; it rediscovers it from the rotation count.

[IMAGE: Stacked horizontal bar of 64 cells representing the coordinate pairs, coloured in three bands: cells 0-20 green labelled "untouched", cells 21-45 amber labelled "ramped", cells 46-63 red labelled "fully interpolated". A second bar directly beneath, from the critical-dimension calculation, marks cells 46-63 as "never completed a rotation in pretraining". The two red regions align exactly. Caption: "YaRN's ramp boundary and the critical dimension are the same boundary, derived two different ways."]

Step 7 — the temperature. With \(s = 8\), the attention scaling is \(\sqrt{1/t} = 0.1 \ln 8 + 1 = 1.208\). Every attention logit is multiplied by 1.208 before the softmax, sharpening a distribution that would otherwise flatten as it spreads over eight times as many keys. Small constant, and without it the extended model's attention entropy drifts upward across the whole context.

You can replay all of it with a calculator, and the payoff is that step 5 and step 6 are the two places where a real extension goes wrong.

Where It Breaks

Perplexity is the wrong instrument, structurally

This is the most expensive misconception in the area, and it is not a matter of insufficient rigour. Next-token prediction is dominated by nearby tokens; the fast pairs that serve those tokens are the ones every extension method protects. So a model can lose its long-range discrimination entirely while its loss curve stays flat, and Men et al. demonstrated exactly that at 32k with retrieval failing at 1k.

Needle-in-a-haystack is better and no longer sufficient on its own. RULER evaluated 17 long-context models across 13 tasks and found that although models claim 32k or more, only about half maintain satisfactory performance at 32k once the tasks include multi-hop tracing and aggregation rather than single-fact lookup (Hsieh et al., 2024, RULER, COLM 2024, arXiv:2404.06654). A single planted fact in an otherwise irrelevant haystack is close to the easiest possible long-context task.

[IMAGE: Two-panel chart over the same x-axis of context length from 1k to 128k. Top panel: validation perplexity, three near-identical flat lines for three base values. Bottom panel: multi-hop retrieval accuracy for the same three models, diverging sharply, with the smallest base collapsing first. Caption: "The instrument everybody watches and the capability everybody claims respond to completely different things."]

The base is a training-time property

Editing rope_theta in a config file and reloading a checkpoint is a common and mostly wrong move. Every key in the cache is now rotated by an angle the weights were never fit against, and the change hits short prompts as hard as long ones. The reason PI, NTK, and YaRN exist as named methods rather than as "set the base higher" is that all three keep the trained angular range intact and change only how positions map into it, and even then all three need at least a short fine-tune at the target length.

Extension does not create dependency

Position augmentation methods such as PoSE simulate long positions inside a short window by adding resampled skipping biases to chunk position indices, avoiding the quadratic attention cost entirely (Zhu et al., 2023, arXiv:2309.10400). They teach the model about unseen indices, which is real and valuable, and they cannot teach it to route information across 100,000 tokens because the attention matrix in training never spanned that far. That is why Llama 3 spent roughly 800B tokens on staged long-context pretraining rather than substituting the cheap thing for the expensive one.

Multi-axis schemes fragment the same budget

Vision and video models partition the head's pairs across temporal, height, and width axes, as Qwen2-VL's M-RoPE does (Wang et al., 2024, arXiv:2409.12191). Every axis then gets a third of the pairs and, critically, a third of the high-frequency pairs that support sharp local attention. Dense-document OCR and hour-long video tracking compete for the same 64 slots, and no setting satisfies both.

The hybrid concentrates the risk

In a 3:1 RoPE/NoPE model, every dependency spanning more than one 4,096-token window flows through the full-attention layers. Those layers are load-bearing in a way no single layer is in a uniform architecture, so uniform pruning or quantisation can destroy long-context behaviour while short-prompt benchmarks stay flat. NoPE's implicit signal is also weaker at exact ordering, so "reverse this list" and "which came first" degrade before "find this fact" does. Meta's decision to pair iRoPE with inference-time attention temperature scaling is a fair signal that the architecture alone does not reach the headline number.

Headline context is capacity, not accuracy

A 10-million-token window means the machinery can accept 10 million tokens. What fraction of them the model can act on is an empirical question with a mostly disappointing answer, and the gap between advertised and effective length has been the stable finding across every generation of models so far.

Alternative Designs

ALiBi deserves a fairer hearing than it usually gets (Press et al., 2021, arXiv:2108.12409). It extrapolates cleanly because a linear penalty is defined at any distance, and it was the honest answer for years. What killed it for frontier use is that the recency prior is baked in and cannot be overridden: a fact 60,000 tokens back is penalised for being 60,000 tokens back no matter how relevant it is, which is precisely the wrong behaviour for retrieval over a long document.

How It Is Used in Practice

The workflow that has settled out looks like this.

Pretrain at a modest length with a base chosen for the final target rather than the pretraining length. This is why Llama 3 pretrained at 8k with base 500,000 rather than the 84,000 its 8k window would have required. The extra spectrum is idle during early training and is what the later stages will need.

Extend in stages with two gates per stage. Short-context evaluation must fully recover, catching the case where extension damaged the fast pairs, and retrieval must be solved at the new length, catching the case where the slow pairs were never usable. Llama 3 ran six such stages over roughly 800B tokens.

Choose the extension method by what you can afford. Full retraining at target length is the most reliable and the most expensive. YaRN is the best-supported cheap option and is implemented in vLLM, Hugging Face Transformers, and most serving stacks as a config-level rope_scaling block. LongRoPE pushes further with a search over per-dimension scaling factors, reporting extension beyond 2 million tokens (Ding et al., 2024, arXiv:2402.13753). PoSE reduces the training cost of whichever mapping you choose.

Evaluate with something that has aggregation and multi-hop tasks in it. RULER is the default; needle-in-a-haystack alone will pass models that cannot do the work.

Operationally, the base and its scaling configuration are a compatibility surface, not a tuning knob. A serving stack that applies a different rope_scaling than the one the checkpoint was tuned under produces a model that loads, runs, generates fluent text, and is subtly wrong, with no error anywhere in the path. It belongs in the same category as tokeniser mismatch: cheap to check, expensive to miss.

[IMAGE: Six-panel strip showing needle-retrieval accuracy heatmaps, x-axis context length, y-axis needle depth, for one model across the six stages of staged context extension from 8k to 128k. Caption: "Each stage buys a band of usable context; the gate is that the previous band stays solid."]

Insights Worth Remembering

1. The base is a spectrum control, not a length setting. It stretches wavelengths from a fixed anchor at \(2\pi\) tokens. Reasoning about it as "bigger base means longer context" gets the mechanism wrong and will mislead you on every second-order question, including why raising it hurts short prompts.

2. There is a floor as well as a ceiling. The received wisdom is that too large a base breaks extrapolation. Too small a base breaks discrimination outright, and the derived floor rises steeply: \(6.4 \times 10^5\) at 32k, \(7.8 \times 10^6\) at 128k.

3. Perplexity cannot see long-context failure, by construction. It is dominated by local tokens, and local tokens are served by the fast pairs that every method protects. A flat loss curve is compatible with total loss of long-range retrieval.

4. The three extension methods differ only in which frequencies they sacrifice. PI sacrifices uniformly, NTK sacrifices the slow end, YaRN sacrifices the slow end while explicitly protecting pairs that complete more than 32 rotations. Once you see the spectrum, they stop being alternatives and become a single design axis.

5. YaRN's ramp rediscovers the critical dimension. Its fully-interpolated group and the set of pairs that never complete a rotation during pretraining are the same eighteen pairs in the worked example. Two derivations, one boundary.

6. RoPE does not work by decay. Barbero et al. prove the expected decay is zero, and show high frequencies build sharp positional heads while low frequencies carry distance-agnostic semantics. Removing the lowest quarter of the spectrum slightly improved validation perplexity on Gemma 2B.

7. Positional encoding stopped being a global choice in 2025. Llama 4 and Command A both apply RoPE to most layers and no positional encoding to the rest. The question is no longer "which scheme" but "which scheme, on which layers, with what attention span".

8. Advertised context is a capacity claim. RULER found roughly half of models claiming 32k or more fail to hold up at 32k once tasks require multi-hop reasoning. Treat the number in the model card as the size of the buffer.

Open Questions

How far below the derived floor can staged training actually carry a model? Measured: Llama 3.1 ships 128k on a base the analysis places at 32k, and it is usable. Unknown: whether that gap is a fixed factor, whether it grows or shrinks with model size, and what precisely the extra training buys in geometric terms. Nobody has published a controlled sweep of base against training-token budget at fixed target length.

Is the high-frequency end underused or overused? Barbero et al. show the lowest frequencies contribute little and that removing them helps slightly at 2B scale on short contexts. They explicitly note they lacked the resources to test long contexts, which is exactly where the low frequencies would matter most. Whether p-RoPE survives at 100k context and 70B parameters is open and testable.

Does the hybrid actually need RoPE at all? If NoPE layers handle long-range retrieval and windowed layers handle local structure, it is not obvious that the windowed layers need rotary embeddings rather than a simpler local scheme. The current 3:1 ratios appear to be empirical choices from limited sweeps rather than derived optima.

Can content-defined position be made to fit a fused kernel? CoPE addresses positions by content rather than token count and solves tasks that defeat every standard scheme (Golovneva et al., 2024, arXiv:2405.18719), at the cost of an \(n \times n\) position tensor that FlashAttention exists to avoid materialising. It is likely that some restricted variant is kernel-compatible; no one has shown which.

What is the right unit for multimodal position? Advancing the index by one after an image asserts that a patch step and a word step are the same displacement. There is no principled conversion, current values are tuning artefacts, and the choice measurably affects interleaved image-text tasks. This is an open modelling question dressed as an implementation detail.

Sources and Further Reading

1. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021). "RoFormer: Enhanced Transformer with Rotary Position Embedding." arXiv:2104.09864
2. Vaswani, A., et al. (2017). "Attention Is All You Need." NeurIPS 2017. arXiv:1706.03762
3. Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). "Self-Attention with Relative Position Representations." NAACL 2018. arXiv:1803.02155
4. Press, O., Smith, N. A., & Lewis, M. (2021). "Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation." ICLR 2022. arXiv:2108.12409
5. Men, X., Xu, M., Wang, B., Zhang, Q., Lin, H., Han, X., & Chen, W. (2024). "Base of RoPE Bounds Context Length." NeurIPS 2024. arXiv:2405.14591
6. Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., & Lin, D. (2023). "Scaling Laws of RoPE-based Extrapolation." ICLR 2024. arXiv:2310.05209
7. Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., & Veličković, P. (2024). "Round and Round We Go! What makes Rotary Positional Encodings useful?" arXiv:2410.06205
8. Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). "Extending Context Window of Large Language Models via Positional Interpolation." arXiv:2306.15595
9. Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023). "YaRN: Efficient Context Window Extension of Large Language Models." ICLR 2024. arXiv:2309.00071
10. Zhu, D., Yang, N., Wang, L., Song, Y., Wu, W., Wei, F., & Li, S. (2023). "PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training." ICLR 2024. arXiv:2309.10400
11. Ding, Y., et al. (2024). "LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens." ICML 2024. arXiv:2402.13753
12. Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., & Reddy, S. (2023). "The Impact of Positional Encoding on Length Generalization in Transformers." NeurIPS 2023. arXiv:2305.19466
13. Yang, B., Venkitesh, B., Talupuru, D., Lin, H., Cairuz, D., Blunsom, P., & Locatelli, A. (2025). "Rope to Nope and Back Again: A New Hybrid Attention Strategy." arXiv:2501.18795
14. Grattafiori, A., et al. (2024). "The Llama 3 Herd of Models." arXiv:2407.21783
15. Rozière, B., et al. (2023). "Code Llama: Open Foundation Models for Code." arXiv:2308.12950
16. Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?" COLM 2024. arXiv:2404.06654
17. Golovneva, O., Wang, T., Weston, J., & Sukhbaatar, S. (2024). "Contextual Position Encoding: Learning to Count What's Important." arXiv:2405.18719
18. Wang, P., et al. (2024). "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution." arXiv:2409.12191
19. Cohere (2025). "Command A: An Enterprise-Ready Large Language Model." Technical report
20. Meta AI (2025). "The Llama 4 herd: the beginning of a new era of natively multimodal AI innovation." Meta AI blog

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.