Embeddings & Representations
The lookup table, the residual stream, contextual vectors, geometry and superposition.
11concepts
47flashcards
81minutes of reading
- 01 Sentence Embeddings and Pooling A transformer produces one vector per token, not one per sentence, so turning a sequence of contextual vectors into a single comparable vector requires a pooling choice that quietly changes what "similar" ends up meaning.
- 02 Static vs Contextual Embeddings The shift from one vector per word to one vector per word-in-context, and why that shift is arguably the single biggest reason transformer-based models outperformed earlier NLP.
- 03 Weight Tying of Input and Output Embeddings Why most language models score every candidate next token using the transpose of the exact same matrix that looked up the input tokens, cutting parameters and improving perplexity in one move.