Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
11concepts
53flashcards
91minutes of reading
- 01 Length Extrapolation What it actually means for a model to handle sequences longer than training, why perplexity is a necessary but misleading metric for it, and how different positional schemes fare with zero adaptation.
- 02 RoPE Scaling: NTK-aware and YaRN How a 4k-token base checkpoint becomes a 128k-token release without pretraining from scratch, by rescaling RoPE's rotation frequencies rather than its raw positions.
- 03 The NoPE Result Decoder-only transformers trained with zero positional encoding of any kind matched or beat RoPE and ALiBi on length generalisation, because the causal mask alone leaks a position signal.