Embeddings & Representations
The lookup table, the residual stream, contextual vectors, geometry and superposition.
17concepts
177flashcards
126minutes of reading
- 01 Binary and Int8 Embedding Quantisation Storing each embedding dimension as one bit instead of thirty-two cuts index memory by 32x and, with a float rescoring pass over the shortlist, gives most of the retrieval quality back.
- 02 Embedding Benchmarks and the Zero-Shot Problem The leaderboard that everyone uses to pick an embedding model publishes training splits for its own test sets, so a top rank increasingly measures in-domain fit rather than the out-of-domain generalisation retrieval actually needs.
- 03 Matryoshka Representation Learning Training an embedding model so that the first m dimensions of every vector are themselves a usable embedding, letting one index be read at 64, 256 or 3072 dimensions without re-encoding the corpus.
- 04 Sentence Embeddings and Pooling A transformer produces one vector per token, not one per sentence, so turning a sequence of contextual vectors into a single comparable vector requires a pooling choice that quietly changes what "similar" ends up meaning.
- 05 Static vs Contextual Embeddings The shift from one vector per word to one vector per word-in-context, and why that shift is arguably the single biggest reason transformer-based models outperformed earlier NLP.
- 06 Weight Tying of Input and Output Embeddings Why most language models score every candidate next token using the transpose of the exact same matrix that looked up the input tokens, cutting parameters and improving perplexity in one move.