TF-IDF and the Vector Space Model

A query for "photosynthesis in the leaf" against a million-document collection matches almost every document on "the" and "in", and roughly one document in a thousand on "photosynthesis". Counting matches ranks an essay that uses "the" four hundred times above the botany paper. Any usable ranking has to decide what a match on each term is worth, and the vector space model was the first framework to make that decision explicit, geometric and tunable (Salton, Wong & Yang, 1975, A Vector Space Model for Automatic Indexing, CACM 18(11)).

Each document \(d\) and query \(q\) becomes a vector with one dimension per vocabulary term, where the component \(w_{t,d}\) records how strongly term \(t\) characterises \(d\). Ranking is by the cosine of the angle between the two vectors:

Nothing in that formula says what the weights should be, which is both the model's strength and its weakness.

Two kinds of evidence, multiplied

A weight answers two questions: how prominent is the term in this document, and how discriminating is it across the collection? The local factor is term frequency \(\mathrm{tf}_{t,d}\), usually damped to \(1 + \log \mathrm{tf}_{t,d}\) because the tenth occurrence of a word is weaker evidence than the second.

The global factor is inverse document frequency. Spärck Jones proposed interpreting term specificity statistically, as a function of how many documents use a term rather than what it means, so that matches on rarer terms count for more (Spärck Jones, 1972, A Statistical Interpretation of Term Specificity and Its Application in Retrieval, Journal of Documentation 28(1)). With \(N\) documents, \(\mathrm{df}_t\) of which contain \(t\):

Put numbers in, with \(N = 10^6\) and base-10 logs. "The", present in 990,000 documents, gets \(\mathrm{idf} \approx 0.004\). "Photosynthesis", in 1,000 documents, gets \(3.0\); "chlorophyll", in 100, gets \(4.0\). A document mentioning photosynthesis four times earns \((1 + 0.60) \times 3.0 \approx 4.8\) on that dimension, while four hundred occurrences of "the" earn \((1 + 2.60) \times 0.004 \approx 0.015\). The stopword problem mostly dissolves without a stop list.

Salton and Buckley's comparison of weighting combinations concluded that appropriately weighted single terms beat more elaborate text representations (Salton & Buckley, 1988, Term-Weighting Approaches in Automatic Text Retrieval, IP&M 24(5)). The SMART system's notation still names schemes by two triplets, one for documents and one for queries: lnc.ltc means log tf, no idf and cosine normalisation on the document side, and log tf, idf and cosine on the query side (Manning, Raghavan & Schütze, 2008, Introduction to Information Retrieval, §6.4).

What normalisation actually does

Dividing by vector length removes the advantage a long document gets simply from containing more distinct terms and higher counts. A document and that same document concatenated with itself point in the same direction and score identically, which is the property one wants.

The trouble is that relevance is not independent of length. Singhal, Buckley and Mitra compared, bucket by bucket of document length, the probability that a document was retrieved with the probability that it was relevant. Cosine normalisation retrieved short documents more often than their relevance warranted and long ones less often, consistently across six collections (Singhal, Buckley & Mitra, 1996, Pivoted Document Length Normalization, SIGIR). Their fix rotates the normaliser around a pivot \(p\), typically the average value of the old normaliser, with slope \(s\):

Documents shorter than the pivot are now divided by more than cosine would use, longer ones by less. With the slope trained on one collection and reused on others, they reported 9 to 12 percent improvements in average precision over plain cosine. The \(b\) parameter of BM25 (see BM25 and Term Frequency Saturation) is the same idea in a different model: partial length normalisation with a tunable slope.

A model, or a well-tuned heuristic?

The vector picture offers no account of why the cosine of weighted counts should track relevance, and critics from the probabilistic tradition said so. Fang, Tao and Zhai made the complaint testable. They wrote down constraints any reasonable ranking function should meet, such as that an extra occurrence of a query term should not lower a score and that, other things equal, a match on a rarer term should count for more, then checked pivoted TF-IDF, BM25 and Dirichlet-smoothed query likelihood against them. None satisfied every constraint unconditionally, and where a function met a constraint only within a parameter range, retrieval quality fell away outside that range (Fang, Tao & Zhai, 2004, A Formal Study of Information Retrieval Heuristics, SIGIR).

Defenders answer that the geometry earns its keep beyond ranking: the same vectors support relevance feedback, clustering and near-duplicate detection.

When it breaks

Terms are orthogonal by construction. "Car" and "automobile" share nothing, and "New York" is two unrelated axes. Vocabulary mismatch is untouched, and it is the gap that query expansion, learned sparse retrieval and dense embeddings each attack.

Library defaults are not the textbook scheme. scikit-learn's TfidfTransformer uses raw counts unless sublinear_tf=True, and by default computes \(\mathrm{idf} = \ln\frac{1+n}{1+\mathrm{df}} + 1\), so a term in every document keeps weight 1. Two papers' "TF-IDF baselines" are often different functions.

IDF belongs to a collection, not to a word. Scores from shards with different document frequencies are not comparable, and a term that becomes common after indexing, such as a product name after launch, stays overweighted until statistics are refreshed.

Length normalisation is collection-specific. A slope tuned on news articles is wrong for product titles, and the length bias returns whenever the corpus changes shape.