Attention Internals
Queries, keys and values, masking, multi-head and grouped-query, sinks, and the quadratic wall.
14concepts
94flashcards
112minutes of reading
- 01 Attention Sinks Why the first few tokens of almost any sequence soak up a disproportionate share of attention regardless of their content, and how that quirk becomes the key to stable infinite-length streaming generation.
- 02 Induction Heads A specific, mechanistically understood attention circuit that copies patterns it has seen once in the current context, and the closest thing the field has to a concrete explanation for in-context learning.
- 03 Linear Attention and Kernel Feature Maps Drop the softmax, replace the exponential kernel with a feature map, and matrix associativity turns quadratic attention into a linear recurrence with constant-size state — plus the quality gap that kept it out of frontier models for five years.
- 04 Multi-Head Latent Attention DeepSeek's answer to the KV cache problem — cache one low-rank latent vector per token instead of every head's keys and values, absorb the up-projection into the query and output weights, and carve out a separate untouched channel for RoPE.
- 05 Multi-Query and Grouped-Query Attention Why the key and value projections, not the query projection, were the ones cut down to fix the real inference bottleneck, and the middle-ground design that most production LLMs settled on.
- 06 Natively Trainable Sparse Attention Most sparse attention is bolted on at inference to a densely trained model, which leaves accuracy on the table and speed unrealised; training sparsity in from the start requires the sparsity pattern to be differentiable and the memory access pattern to suit a GPU.