Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
11concepts
53flashcards
91minutes of reading
- 01 Learned Absolute Positions The simplest positional scheme, a trainable embedding table indexed by slot number, and the hard ceiling it builds into every model that uses it.
- 02 Why Attention Needs Positions Self-attention is a permutation-equivariant set operation by default; every notion of word order a transformer has was injected as data, not built into the architecture.
- 03 ALiBi: Attention with Linear Biases A zero-parameter alternative to rotating or embedding position, a per-head linear penalty on distance, and why train-short-test-long comes at the cost of sharp long-range recall.
- 04 Effective vs Nominal Context Length The advertised context window and the length a model actually uses well are different numbers, often by a large factor, and the gap between them has several distinct, measurable causes.
- 05 Positional Encodings Why attention needs to be told where each token sits, and how RoPE, ALiBi, and the surprising NoPE result shape how far a model can read.
- 06 Relative Position Representations The 2018 idea that attention should see i-j directly rather than infer it from two absolute positions, the direct ancestor of both RoPE and T5's relative bias.
- 07 Rotary Position Embeddings (RoPE) The rotation trick behind Llama, Mistral, and Qwen, worked through in full, and why an exact algebraic guarantee beats hoping the network learns relative distance on its own.
- 08 Sinusoidal Positional Encodings The original transformer's fix for order, a fixed sin/cos signal added to every embedding, and why its elegant closed form still degrades once you run past training length.
- 09 Length Extrapolation What it actually means for a model to handle sequences longer than training, why perplexity is a necessary but misleading metric for it, and how different positional schemes fare with zero adaptation.
- 10 RoPE Scaling: NTK-aware and YaRN How a 4k-token base checkpoint becomes a 128k-token release without pretraining from scratch, by rescaling RoPE's rotation frequencies rather than its raw positions.
- 11 The NoPE Result Decoder-only transformers trained with zero positional encoding of any kind matched or beat RoPE and ALiBi on length generalisation, because the causal mask alone leaks a position signal.