Attention Internals
Queries, keys and values, masking, multi-head and grouped-query, sinks, and the quadratic wall.
11concepts
49flashcards
88minutes of reading
- 01 Attention Mechanism How attention lets a model focus on the relevant parts of a sequence by computing weighted dependencies between every pair of positions.
- 02 Causal Masking How a transformer trained to predict every position in parallel is stopped from cheating by looking ahead, and why the mask is applied before softmax rather than after.
- 03 Multi-Head Attention in Depth Why attention runs as several small parallel attentions rather than one large one, what each head actually gets to specialise in, and why most heads turn out to be redundant.
- 04 Queries, Keys, and Values The retrieval metaphor behind attention, made literal - what a query, key, and value actually are as learned projections, and why three separate matrices instead of one.
- 05 Self-Attention vs Cross-Attention The one-line distinction, where the queries come from versus where the keys and values come from, and why decoder-only LLMs mostly made cross-attention disappear from the mainstream architecture.
- 06 Sliding-Window Attention The simplest fix to attention's quadratic cost - cap how far back each position can look - and why stacking layers gets that fixed window back to an effectively long range.
- 07 The Quadratic Cost of Attention Where the O(n squared) in "attention is quadratic" actually comes from, why it hits both compute and memory, and what doubling the context length really costs.