Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
16concepts
157flashcards
128minutes of reading
- 01 Learned Absolute Positions The simplest positional scheme, a trainable embedding table indexed by slot number, and the hard ceiling it builds into every model that uses it.
- 02 Why Attention Needs Positions Self-attention is a permutation-equivariant set operation by default; every notion of word order a transformer has was injected as data, not built into the architecture.