Embeddings & Representations
The lookup table, the residual stream, contextual vectors, geometry and superposition.
17concepts
177flashcards
126minutes of reading
- 01 Cosine Similarity vs Dot Product Two near-identical looking formulas that answer different questions, one measures direction alone, the other measures direction and magnitude together, and picking the wrong one silently breaks a search or ranking system.
- 02 Embeddings and Semantic Search How dense vectors turn text into a geometry of meaning, and how cosine similarity lets you find related content without keywords.
- 03 Input Embeddings and the Lookup Table How a token id becomes a vector, a single row lookup into a trained matrix, and why that matrix is often the largest single block of parameters a small model spends before any real computation happens.
- 04 word2vec and GloVe The two ideas, predict-a-neighbour and factor-a-co-occurrence-matrix, that first proved dense vectors could capture enough word meaning to support arithmetic on it, years before transformers existed.
- 05 Binary and Int8 Embedding Quantisation Storing each embedding dimension as one bit instead of thirty-two cuts index memory by 32x and, with a float rescoring pass over the shortlist, gives most of the retrieval quality back.
- 06 Embedding Benchmarks and the Zero-Shot Problem The leaderboard that everyone uses to pick an embedding model publishes training splits for its own test sets, so a top rank increasingly measures in-domain fit rather than the out-of-domain generalisation retrieval actually needs.
- 07 Matryoshka Representation Learning Training an embedding model so that the first m dimensions of every vector are themselves a usable embedding, letting one index be read at 64, 256 or 3072 dimensions without re-encoding the corpus.
- 08 Sentence Embeddings and Pooling A transformer produces one vector per token, not one per sentence, so turning a sequence of contextual vectors into a single comparable vector requires a pooling choice that quietly changes what "similar" ends up meaning.
- 09 Static vs Contextual Embeddings The shift from one vector per word to one vector per word-in-context, and why that shift is arguably the single biggest reason transformer-based models outperformed earlier NLP.
- 10 Weight Tying of Input and Output Embeddings Why most language models score every candidate next token using the transpose of the exact same matrix that looked up the input tokens, cutting parameters and improving perplexity in one move.
- 11 Cross-Lingual Embedding Alignment Putting a hundred languages into one vector space so that a Hindi query retrieves an English document requires an explicit alignment signal, and the way you supply it determines exactly how the space fails.
- 12 Embedding Geometry and Anisotropy Learned embedding spaces are not the well-spread sphere the geometric intuition suggests, they collapse into a narrow cone, and that fact quietly breaks naive similarity comparisons built on top of them.
- 13 Hard Negative Mining and Contrastive Embedding Training A retrieval embedder is only as good as the negatives it was trained against, and the gap between easy in-batch negatives and mined hard negatives is the single largest lever in dual-encoder training.
- 14 Hubness in High-Dimensional Retrieval In high-dimensional spaces a small number of points appear in almost everyone's nearest-neighbour list regardless of relevance, which is a property of the geometry rather than a bug in the embedder.
- 15 Superposition and Polysemantic Neurons Individual neurons in a trained network routinely fire for several unrelated concepts at once, and the leading explanation is not noise, it is a model deliberately packing more features than it has dimensions using near-orthogonal directions.
- 16 The Residual Stream Reframing a transformer's residual connections as one shared, additive vector space that every layer reads from and writes to, the lens that makes attention, MLPs, and interpretability results legible as a single system.
- 17 Unembedding and the Logit Lens Multiplying an intermediate layer's residual stream by the final unembedding matrix, as if it were the last layer, turns out to produce surprisingly sensible next-token guesses, a cheap window into what a model has committed to mid-computation.