Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
16concepts
157flashcards
128minutes of reading
- 01 Learned Absolute Positions The simplest positional scheme, a trainable embedding table indexed by slot number, and the hard ceiling it builds into every model that uses it.
- 02 Why Attention Needs Positions Self-attention is a permutation-equivariant set operation by default; every notion of word order a transformer has was injected as data, not built into the architecture.
- 03 ALiBi: Attention with Linear Biases A zero-parameter alternative to rotating or embedding position, a per-head linear penalty on distance, and why train-short-test-long comes at the cost of sharp long-range recall.
- 04 Effective vs Nominal Context Length The advertised context window and the length a model actually uses well are different numbers, often by a large factor, and the gap between them has several distinct, measurable causes.
- 05 Multidimensional RoPE for Images and Video A video is three coordinates, not one, and flattening it into a token index throws away the fact that two patches were vertically adjacent; M-RoPE and 2D RoPE split the head dimension into independent axes so a single mechanism can encode time, height, and width.
- 06 Positional Encodings Why attention needs to be told where each token sits, and how RoPE, ALiBi, and the surprising NoPE result shape how far a model can read.
- 07 Relative Position Representations The 2018 idea that attention should see i-j directly rather than infer it from two absolute positions, the direct ancestor of both RoPE and T5's relative bias.
- 08 Rotary Position Embeddings (RoPE) The rotation trick behind Llama, Mistral, and Qwen, worked through in full, and why an exact algebraic guarantee beats hoping the network learns relative distance on its own.
- 09 Sinusoidal Positional Encodings The original transformer's fix for order, a fixed sin/cos signal added to every embedding, and why its elegant closed form still degrades once you run past training length.
- 10 Contextual Position Encoding Every standard position scheme counts tokens, which is why a model cannot reliably attend to "the previous sentence"; CoPE makes the position counter itself a function of content, incrementing only on tokens the model decides matter.
- 11 Interleaving RoPE and NoPE Layers Llama 4 and Cohere's Command A both build long-context models by using rotary embeddings on most layers and no positional encoding at all on the rest, turning a choice everyone treated as global into a per-layer one.
- 12 Length Extrapolation What it actually means for a model to handle sequences longer than training, why perplexity is a necessary but misleading metric for it, and how different positional schemes fare with zero adaptation.
- 13 Position-Augmented Training If a model fails at length because it has never seen large position indices, the cheapest fix is not a longer training sequence but a shuffled or skipped one, showing the model position 90,000 inside a 2,048-token batch.
- 14 RoPE Scaling: NTK-aware and YaRN How a 4k-token base checkpoint becomes a 128k-token release without pretraining from scratch, by rescaling RoPE's rotation frequencies rather than its raw positions.
- 15 The NoPE Result Decoder-only transformers trained with zero positional encoding of any kind matched or beat RoPE and ALiBi on length generalisation, because the causal mask alone leaks a position signal.
- 16 The RoPE Base Frequency One hyperparameter, written as 10000 in the original RoFormer code and 500000 in Llama 3, sets the entire wavelength spectrum of rotary position embedding and puts a hard ceiling on the context length the model can actually discriminate.