Attention Internals
Queries, keys and values, masking, multi-head and grouped-query, sinks, and the quadratic wall.
11concepts
49flashcards
88minutes of reading
- 01 Attention as Soft Dictionary Lookup The mental model that makes attention click before the matrix algebra does - a lookup table where the key match is a matter of degree, not an exact hit or miss.
- 02 Attention Mechanism How attention lets a model focus on the relevant parts of a sequence by computing weighted dependencies between every pair of positions.
- 03 Causal Masking How a transformer trained to predict every position in parallel is stopped from cheating by looking ahead, and why the mask is applied before softmax rather than after.
- 04 Multi-Head Attention in Depth Why attention runs as several small parallel attentions rather than one large one, what each head actually gets to specialise in, and why most heads turn out to be redundant.
- 05 Queries, Keys, and Values The retrieval metaphor behind attention, made literal - what a query, key, and value actually are as learned projections, and why three separate matrices instead of one.
- 06 Self-Attention vs Cross-Attention The one-line distinction, where the queries come from versus where the keys and values come from, and why decoder-only LLMs mostly made cross-attention disappear from the mainstream architecture.
- 07 Sliding-Window Attention The simplest fix to attention's quadratic cost - cap how far back each position can look - and why stacking layers gets that fixed window back to an effectively long range.
- 08 The Quadratic Cost of Attention Where the O(n squared) in "attention is quadratic" actually comes from, why it hits both compute and memory, and what doubling the context length really costs.
- 09 Attention Sinks Why the first few tokens of almost any sequence soak up a disproportionate share of attention regardless of their content, and how that quirk becomes the key to stable infinite-length streaming generation.
- 10 Induction Heads A specific, mechanistically understood attention circuit that copies patterns it has seen once in the current context, and the closest thing the field has to a concrete explanation for in-context learning.
- 11 Multi-Query and Grouped-Query Attention Why the key and value projections, not the query projection, were the ones cut down to fix the real inference bottleneck, and the middle-ground design that most production LLMs settled on.