Transformer Anatomy
The block, the stack, encoder vs decoder, MoE, and the design choices that separate model families.
14concepts
58flashcards
113minutes of reading
- 01 Mixture of Experts Why MoE models can be 10x cheaper to serve than dense models of the same capability, and what makes them hard to train.
- 02 Prefix Language Models A single decoder-only stack that grants bidirectional attention to a prefix segment before switching to causal generation, and why the field mostly passed on this compromise anyway.
- 03 QK-Normalisation Normalising queries and keys before the attention dot product to stop logits from blowing up at scale, the fix that made 22-billion-parameter and larger transformers trainable without loss spikes.
- 04 Weight Initialisation in Transformers Why the last matrix in every attention and FFN sublayer gets shrunk by 1/sqrt(2N) at initialisation, and the more principled framework, muP, that turns hyperparameter tuning at scale into a lookup instead of a search.