The RoPE Base Frequency

It stretches the slow end while leaving the fast end fixed. Pair \(i\) has wavelength \(\lambda_i = 2\pi b^{2i/d}\), so pair 0 always has \(\lambda_0 = 2\pi \approx 6.3\) tokens no matter what the base is. For a 128-dim head, the slowest pair moves from about 54,400 tokens at \(b=10^4\) to about 2.56 million at \(b=5 \times 10^5\). The base fans the spectrum out from a fixed anchor rather than shifting it.

It is the index separating pairs whose wavelength is shorter than the training length from those whose wavelength is longer. Pairs below it sweep complete periods during training and have seen every angle; pairs above it have only seen one short arc. At inference beyond the training length, those slow pairs are queried at angles never observed, which is the out-of-distribution failure. With \(b=10^4\), \(d=128\), \(L=4096\), forty-six of sixty-four pairs are fully trained and eighteen are not.

The RoPE base. Men et al. derive a lower bound: below it the quantity \(\sum_i \cos(m\theta_i)\) goes negative at some distance, meaning random tokens outscore similar ones and long-range discrimination is destroyed. Their numerical solution requires roughly \(6.4 \times 10^5\) for 32k and \(7.8 \times 10^6\) for 128k. Perplexity is dominated by local tokens and stays healthy throughout, so only retrieval evaluation exposes it.

The base is a training-time property. Changing it re-rotates every key by angles the model's weights were never fit against, so attention geometry shifts everywhere at once, including on short prompts where the model was working. That is why deployment-time extension uses interpolation schemes such as Position Interpolation or YaRN, which map long positions back into the trained angular range, and why any of them still needs a short fine-tune at the target length.

The standard justification is that RoPE decays attention with distance. Barbero et al. show that with Gaussian queries and keys the expected decay is zero, so decay cannot be the mechanism. What they observe instead: the highest frequencies build sharp positional patterns like diagonal and previous-token heads, provably impossible without positional information, while the lowest frequencies are preferred for carrying semantic content that should be distance-agnostic.

Staged long-context pretraining rather than a bigger base. Llama 3 raised context in six stages from 8K to 128K over roughly 800B tokens, advancing only when short-context evals had fully recovered and needle-in-a-haystack was solved at that length. The 500,000 value was validated for lengths up to 32,768, which matches Men et al.'s independently derived 32k floor almost exactly. Base sets what is representable; training decides what is usable.

It stretches the slow end while leaving the fast end fixed. Pair \(i\) has wavelength \(\lambda_i = 2\pi b^{2i/d}\), so pair 0 always has \(\lambda_0 = 2\pi \approx 6.3\) tokens no matter what the base is. For a 128-dim head, the slowest pair moves from about 54,400 tokens at \(b=10^4\) to about 2.56 million at \(b=5 \times 10^5\). The base fans the spectrum out from a fixed anchor rather than shifting it.

It is the index separating pairs whose wavelength is shorter than the training length from those whose wavelength is longer. Pairs below it sweep complete periods during training and have seen every angle; pairs above it have only seen one short arc. At inference beyond the training length, those slow pairs are queried at angles never observed, which is the out-of-distribution failure. With \(b=10^4\), \(d=128\), \(L=4096\), forty-six of sixty-four pairs are fully trained and eighteen are not.

The RoPE base. Men et al. derive a lower bound: below it the quantity \(\sum_i \cos(m\theta_i)\) goes negative at some distance, meaning random tokens outscore similar ones and long-range discrimination is destroyed. Their numerical solution requires roughly \(6.4 \times 10^5\) for 32k and \(7.8 \times 10^6\) for 128k. Perplexity is dominated by local tokens and stays healthy throughout, so only retrieval evaluation exposes it.

The base is a training-time property. Changing it re-rotates every key by angles the model's weights were never fit against, so attention geometry shifts everywhere at once, including on short prompts where the model was working. That is why deployment-time extension uses interpolation schemes such as Position Interpolation or YaRN, which map long positions back into the trained angular range, and why any of them still needs a short fine-tune at the target length.

The standard justification is that RoPE decays attention with distance. Barbero et al. show that with Gaussian queries and keys the expected decay is zero, so decay cannot be the mechanism. What they observe instead: the highest frequencies build sharp positional patterns like diagonal and previous-token heads, provably impossible without positional information, while the lowest frequencies are preferred for carrying semantic content that should be distance-agnostic.

Staged long-context pretraining rather than a bigger base. Llama 3 raised context in six stages from 8K to 128K over roughly 800B tokens, advancing only when short-context evals had fully recovered and needle-in-a-haystack was solved at that length. The 500,000 value was validated for lengths up to 32,768, which matches Men et al.'s independently derived 32k floor almost exactly. Base sets what is representable; training decides what is usable.