Attention Internals
Queries, keys and values, masking, multi-head and grouped-query, sinks, and the quadratic wall.
14concepts
94flashcards
112minutes of reading
- 01 Attention as Soft Dictionary Lookup The mental model that makes attention click before the matrix algebra does - a lookup table where the key match is a matter of degree, not an exact hit or miss.
- 02 Attention Mechanism How attention lets a model focus on the relevant parts of a sequence by computing weighted dependencies between every pair of positions.
- 03 Causal Masking How a transformer trained to predict every position in parallel is stopped from cheating by looking ahead, and why the mask is applied before softmax rather than after.
- 04 Multi-Head Attention in Depth Why attention runs as several small parallel attentions rather than one large one, what each head actually gets to specialise in, and why most heads turn out to be redundant.
- 05 Queries, Keys, and Values The retrieval metaphor behind attention, made literal - what a query, key, and value actually are as learned projections, and why three separate matrices instead of one.
- 06 Self-Attention vs Cross-Attention The one-line distinction, where the queries come from versus where the keys and values come from, and why decoder-only LLMs mostly made cross-attention disappear from the mainstream architecture.
- 07 Sliding-Window Attention The simplest fix to attention's quadratic cost - cap how far back each position can look - and why stacking layers gets that fixed window back to an effectively long range.
- 08 The Quadratic Cost of Attention Where the O(n squared) in "attention is quadratic" actually comes from, why it hits both compute and memory, and what doubling the context length really costs.
- 09 Attention Sinks Why the first few tokens of almost any sequence soak up a disproportionate share of attention regardless of their content, and how that quirk becomes the key to stable infinite-length streaming generation.
- 10 Induction Heads A specific, mechanistically understood attention circuit that copies patterns it has seen once in the current context, and the closest thing the field has to a concrete explanation for in-context learning.
- 11 Linear Attention and Kernel Feature Maps Drop the softmax, replace the exponential kernel with a feature map, and matrix associativity turns quadratic attention into a linear recurrence with constant-size state — plus the quality gap that kept it out of frontier models for five years.
- 12 Multi-Head Latent Attention DeepSeek's answer to the KV cache problem — cache one low-rank latent vector per token instead of every head's keys and values, absorb the up-projection into the query and output weights, and carve out a separate untouched channel for RoPE.
- 13 Multi-Query and Grouped-Query Attention Why the key and value projections, not the query projection, were the ones cut down to fix the real inference bottleneck, and the middle-ground design that most production LLMs settled on.
- 14 Natively Trainable Sparse Attention Most sparse attention is bolted on at inference to a densely trained model, which leaves accuracy on the table and speed unrealised; training sparsity in from the start requires the sparsity pattern to be differentiable and the memory access pattern to suit a GPU.