Transformer Anatomy
The block, the stack, encoder vs decoder, MoE, and the design choices that separate model families.
14concepts
58flashcards
113minutes of reading
- 01 Decoder-Only and Why It Won Why a single causal stack with one unified training objective beat two-stack and bidirectional designs at scale, and the bidirectional context it deliberately gives up to get there.
- 02 Depth, Width, and Aspect Ratio Given a fixed parameter budget, why pretraining loss barely cares whether you go deep or wide, and the hardware and capability reasons architects still don't pick the ratio at random.
- 03 Encoder-Decoder Models (T5) T5's text-to-text framing and the extra cross-attention sublayer that lets a decoder condition on a separately-encoded input, and the real cost/benefit case for keeping two stacks instead of one.
- 04 Encoder-Only Models (BERT) BERT's bidirectional masked-language-model objective, why it makes phenomenal embeddings and classifiers, and why that same bidirectionality makes it structurally unable to generate open-ended text.
- 05 Feed-Forward Networks and SwiGLU The other half of every transformer block, where most of the parameters and arguably most of the stored knowledge live, and why the activation function quietly became SwiGLU.
- 06 Layer Normalization and Residual Connections The two pieces of transformer plumbing that make deep stacks trainable at all, why pre-norm beat post-norm, and how RMSNorm shaved the design down further.
- 07 Parallel Attention and FFN Computing attention and the feed-forward sublayer from the same normalised input instead of sequentially, trading a small representational restriction for less synchronisation on the way to more throughput at scale.
- 08 Sub-Layer Ordering and Design Choices The menu of ordering decisions inside and across transformer blocks beyond pre-norm versus post-norm, from where the final normalisation sits to why attention always runs before the feed-forward sublayer.
- 09 Transformer Architecture The encoder-decoder stack that replaced recurrence and powered every modern LLM.