Classical Information Retrieval
Inverted indexes, TF-IDF and BM25, query processing, and the lexical baselines that refuse to be beaten.
7concepts
141flashcards
51minutes of reading
- 01 Text Analysis: Stemming, Lemmatisation and Stopwords How an analyzer chain turns raw text into index terms, what stemming, lemmatisation and stop lists trade between recall and precision, and why these decisions are fixed at indexing time.
- 02 BM25 and Term Frequency Saturation Why a term appearing twenty times should not score ten times higher than one appearing twice, how BM25 encodes that as a saturating function, and what its two parameters actually control.
- 03 Inverted Indexes and Postings Lists The data structure that makes text search sublinear in corpus size, why postings are stored as sorted document IDs, and how gap encoding turns a list of integers into a few bits each.
- 04 TF-IDF and the Vector Space Model Queries and documents as weighted term vectors ranked by cosine, why each weight multiplies local and collection-wide evidence, and why plain cosine normalisation systematically favours short documents.
- 05 Dynamic Pruning with WAND and Block-Max Retrieving the exact top-k without scoring most of the candidates, by maintaining an upper bound on what each document could score and skipping everything that cannot beat the current threshold.
- 06 Learned Sparse Retrieval Using a language model to assign weights over the vocabulary, including terms not present in the text, so semantic matching runs on an inverted index instead of a vector index.
- 07 Query Likelihood Language Models for Retrieval Ranking documents by the probability that each document's language model generates the query, and why the smoothing that makes this work quietly reintroduces IDF weighting and length normalisation.