Transformer Anatomy
The block, the stack, encoder vs decoder, MoE, and the design choices that separate model families.
14concepts
58flashcards
113minutes of reading
- 01 Anatomy of a Transformer Block The exact sequence of operations inside one transformer block, from tensor shapes to parameter counts, and why every frontier model is just this same function stacked dozens of times.
- 02 Decoder-Only and Why It Won Why a single causal stack with one unified training objective beat two-stack and bidirectional designs at scale, and the bidirectional context it deliberately gives up to get there.
- 03 Depth, Width, and Aspect Ratio Given a fixed parameter budget, why pretraining loss barely cares whether you go deep or wide, and the hardware and capability reasons architects still don't pick the ratio at random.
- 04 Encoder-Decoder Models (T5) T5's text-to-text framing and the extra cross-attention sublayer that lets a decoder condition on a separately-encoded input, and the real cost/benefit case for keeping two stacks instead of one.
- 05 Encoder-Only Models (BERT) BERT's bidirectional masked-language-model objective, why it makes phenomenal embeddings and classifiers, and why that same bidirectionality makes it structurally unable to generate open-ended text.
- 06 Feed-Forward Networks and SwiGLU The other half of every transformer block, where most of the parameters and arguably most of the stored knowledge live, and why the activation function quietly became SwiGLU.
- 07 Layer Normalization and Residual Connections The two pieces of transformer plumbing that make deep stacks trainable at all, why pre-norm beat post-norm, and how RMSNorm shaved the design down further.
- 08 Parallel Attention and FFN Computing attention and the feed-forward sublayer from the same normalised input instead of sequentially, trading a small representational restriction for less synchronisation on the way to more throughput at scale.
- 09 Sub-Layer Ordering and Design Choices The menu of ordering decisions inside and across transformer blocks beyond pre-norm versus post-norm, from where the final normalisation sits to why attention always runs before the feed-forward sublayer.
- 10 Transformer Architecture The encoder-decoder stack that replaced recurrence and powered every modern LLM.
- 11 Mixture of Experts Why MoE models can be 10x cheaper to serve than dense models of the same capability, and what makes them hard to train.
- 12 Prefix Language Models A single decoder-only stack that grants bidirectional attention to a prefix segment before switching to causal generation, and why the field mostly passed on this compromise anyway.
- 13 QK-Normalisation Normalising queries and keys before the attention dot product to stop logits from blowing up at scale, the fix that made 22-billion-parameter and larger transformers trainable without loss spikes.
- 14 Weight Initialisation in Transformers Why the last matrix in every attention and FFN sublayer gets shrunk by 1/sqrt(2N) at initialisation, and the more principled framework, muP, that turns hyperparameter tuning at scale into a lookup instead of a search.