Embeddings & Representations
The lookup table, the residual stream, contextual vectors, geometry and superposition.
17concepts
177flashcards
126minutes of reading
- 01 Cross-Lingual Embedding Alignment Putting a hundred languages into one vector space so that a Hindi query retrieves an English document requires an explicit alignment signal, and the way you supply it determines exactly how the space fails.
- 02 Embedding Geometry and Anisotropy Learned embedding spaces are not the well-spread sphere the geometric intuition suggests, they collapse into a narrow cone, and that fact quietly breaks naive similarity comparisons built on top of them.
- 03 Hard Negative Mining and Contrastive Embedding Training A retrieval embedder is only as good as the negatives it was trained against, and the gap between easy in-batch negatives and mined hard negatives is the single largest lever in dual-encoder training.
- 04 Hubness in High-Dimensional Retrieval In high-dimensional spaces a small number of points appear in almost everyone's nearest-neighbour list regardless of relevance, which is a property of the geometry rather than a bug in the embedder.
- 05 Superposition and Polysemantic Neurons Individual neurons in a trained network routinely fire for several unrelated concepts at once, and the leading explanation is not noise, it is a model deliberately packing more features than it has dimensions using near-orthogonal directions.
- 06 The Residual Stream Reframing a transformer's residual connections as one shared, additive vector space that every layer reads from and writes to, the lens that makes attention, MLPs, and interpretability results legible as a single system.
- 07 Unembedding and the Logit Lens Multiplying an intermediate layer's residual stream by the final unembedding matrix, as if it were the last layer, turns out to produce surprisingly sensible next-token guesses, a cheap window into what a model has committed to mid-computation.