Portfolio ↗

Attention Internals

Queries, keys and values, masking, multi-head and grouped-query, sinks, and the quadratic wall.

11concepts
49flashcards
88minutes of reading
Start with Attention as Soft Dictionary Lookup Drill the deck
All levels beginner intermediate advanced
  1. 01 Attention as Soft Dictionary Lookup The mental model that makes attention click before the matrix algebra does - a lookup table where the key match is a matter of degree, not an exact hit or miss. beginner 6m 5 cards