Classical Information Retrieval
Inverted indexes, TF-IDF and BM25, query processing, and the lexical baselines that refuse to be beaten.
7concepts
141flashcards
51minutes of reading
- 01 Dynamic Pruning with WAND and Block-Max Retrieving the exact top-k without scoring most of the candidates, by maintaining an upper bound on what each document could score and skipping everything that cannot beat the current threshold.
- 02 Learned Sparse Retrieval Using a language model to assign weights over the vocabulary, including terms not present in the text, so semantic matching runs on an inverted index instead of a vector index.
- 03 Query Likelihood Language Models for Retrieval Ranking documents by the probability that each document's language model generates the query, and why the smoothing that makes this work quietly reintroduces IDF weighting and length normalisation.