Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
16concepts
157flashcards
128minutes of reading
- 01 ALiBi: Attention with Linear Biases A zero-parameter alternative to rotating or embedding position, a per-head linear penalty on distance, and why train-short-test-long comes at the cost of sharp long-range recall.
- 02 Effective vs Nominal Context Length The advertised context window and the length a model actually uses well are different numbers, often by a large factor, and the gap between them has several distinct, measurable causes.
- 03 Multidimensional RoPE for Images and Video A video is three coordinates, not one, and flattening it into a token index throws away the fact that two patches were vertically adjacent; M-RoPE and 2D RoPE split the head dimension into independent axes so a single mechanism can encode time, height, and width.
- 04 Positional Encodings Why attention needs to be told where each token sits, and how RoPE, ALiBi, and the surprising NoPE result shape how far a model can read.
- 05 Relative Position Representations The 2018 idea that attention should see i-j directly rather than infer it from two absolute positions, the direct ancestor of both RoPE and T5's relative bias.
- 06 Rotary Position Embeddings (RoPE) The rotation trick behind Llama, Mistral, and Qwen, worked through in full, and why an exact algebraic guarantee beats hoping the network learns relative distance on its own.
- 07 Sinusoidal Positional Encodings The original transformer's fix for order, a fixed sin/cos signal added to every embedding, and why its elegant closed form still degrades once you run past training length.