Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
16concepts
157flashcards
128minutes of reading
- 01 Contextual Position Encoding Every standard position scheme counts tokens, which is why a model cannot reliably attend to "the previous sentence"; CoPE makes the position counter itself a function of content, incrementing only on tokens the model decides matter.
- 02 Interleaving RoPE and NoPE Layers Llama 4 and Cohere's Command A both build long-context models by using rotary embeddings on most layers and no positional encoding at all on the rest, turning a choice everyone treated as global into a per-layer one.
- 03 Length Extrapolation What it actually means for a model to handle sequences longer than training, why perplexity is a necessary but misleading metric for it, and how different positional schemes fare with zero adaptation.
- 04 Position-Augmented Training If a model fails at length because it has never seen large position indices, the cheapest fix is not a longer training sequence but a shuffled or skipped one, showing the model position 90,000 inside a 2,048-token batch.
- 05 RoPE Scaling: NTK-aware and YaRN How a 4k-token base checkpoint becomes a 128k-token release without pretraining from scratch, by rescaling RoPE's rotation frequencies rather than its raw positions.
- 06 The NoPE Result Decoder-only transformers trained with zero positional encoding of any kind matched or beat RoPE and ALiBi on length generalisation, because the causal mask alone leaks a position signal.
- 07 The RoPE Base Frequency One hyperparameter, written as 10000 in the original RoFormer code and 500000 in Llama 3, sets the entire wavelength spectrum of rotary position embedding and puts a hard ceiling on the context length the model can actually discriminate.