Flashcards
11,105 cards in 101 decks, written from the same source as the concepts. Space to flip, arrows to move, K and R to sort what you know from what you do not. Progress lives in your browser.
Transformer Internals
Open the box. How a language model actually turns text into predictions.
Tokenisation
BPE, WordPiece, Unigram, and the ways subword vocabularies quietly shape model behaviour.
Embeddings & Representations
The lookup table, the residual stream, contextual vectors, geometry and superposition.
Attention Internals
Queries, keys and values, masking, multi-head and grouped-query, sinks, and the quadratic wall.
Positional Encoding
Sinusoidal, learned, RoPE, ALiBi, and how context windows get stretched past training length.
Transformer Anatomy
The block, the stack, encoder vs decoder, MoE, and the design choices that separate model families.
Training Objectives
Next-token prediction, masked LM, span corruption, fill-in-the-middle, and auxiliary losses.
Decoding & Generation
Greedy, beam, temperature, top-k, nucleus, and constrained generation into structured formats.
Context & In-Context Learning
Autoregressive generation, the prompt stack, context engineering, and long-context degradation.