Query Likelihood Language Models for Retrieval

Take a 5,000-word article that contains three of the four words in a query. Estimate a unigram model from it by counting, and the probability of the missing word is zero, so the probability of the whole query is zero. The article ties with a page about nothing at all. Fixing that zero is the entire craft of the language-modelling approach to retrieval, and the fix turns out to rebuild, from probability theory, the heuristics that vector-space systems had tuned by hand.

Ponte and Croft proposed ranking documents by how likely a model estimated from each document was to generate the query, folding indexing and retrieval into a single probabilistic model, and reported that it significantly outperformed standard tf-idf weighting on two collections (Ponte & Croft, 1998, A Language Modeling Approach to Information Retrieval, SIGIR). The form that became standard is the multinomial one. With \(c(t,q)\) the count of term \(t\) in the query:

Two smoothing methods

Smoothing mixes the document's maximum-likelihood estimate with a collection model \(P(t \mid C)\), the term's frequency across the whole corpus.

Jelinek-Mercer interpolates with a fixed weight \(\lambda\):

Dirichlet prior smoothing treats the collection model as \(\mu\) pseudo-counts added to the document:

The difference matters. Jelinek-Mercer smooths every document by the same fraction. Dirichlet smooths short documents heavily and long ones lightly, which is the right behaviour for an estimation problem: a long document is a larger sample, so its own counts deserve more trust.

Where IDF comes from

Zhai and Lafferty showed that any smoothing of this shape gives a ranking formula with recognisable parts. Let \(P_s(t \mid d)\) be the smoothed probability for terms that appear in \(d\), and \(\alpha_d\, P(t \mid C)\) the probability for terms that do not. Then

after dropping a term that is identical for every document. Dividing by \(P(t \mid C)\) rewards matches on rare terms, which they note plays a role very similar to IDF, and \(|q| \log \alpha_d\) is a document-length adjustment (Zhai & Lafferty, 2001, A Study of Smoothing Methods for Language Models Applied to Ad Hoc Information Retrieval, SIGIR). For Dirichlet smoothing this becomes

Now the numbers. With \(\mu = 2000\) and a term occurring three times in a document, a rare term with \(P(t \mid C) = 10^{-5}\) contributes \(\ln(1 + 3/0.02) = \ln 151 \approx 5.0\). A common term with \(P(t \mid C) = 0.01\) contributes \(\ln(1 + 3/20) \approx 0.14\). For a two-term query, the length component is \(2 \ln(2000/2500) \approx -0.45\) for a 500-word document and \(2 \ln(2000/7000) \approx -2.5\) for a 5,000-word one. The long document must earn two extra nats from its matches to break even, which is length normalisation derived rather than designed.

What the parameters mean, and the argument about them

Zhai and Lafferty found the optimal \(\mu\) varied from collection to collection but was around 2,000 in most cases, and that the best Jelinek-Mercer \(\lambda\) depended strongly on query type: around 0.1 for short title queries and around 0.7 for long, verbose ones. Their explanation is that smoothing does two jobs. It improves the estimate of the document model, which Dirichlet does well, and it explains away the common, non-informative words that fill verbose queries, which heavy interpolation does well. Dirichlet smoothing did much better on title queries than on verbose ones; Jelinek-Mercer was most effective when queries were verbose. The journal version also evaluates a two-stage method that combines both (Zhai & Lafferty, 2004, TOIS 22(2)).

The "use \(\mu = 2000\)" folklore deserves suspicion. Fang, Tao and Zhai, tuning the same Dirichlet formula for average precision across seven collections and four query types, found optimal values from 800 to 20,000 (Fang, Tao & Zhai, 2004, SIGIR). The broader claim that language models are more principled than BM25 is also contested: most of their practical strength comes from smoothing, and smoothing is exactly where the old heuristics re-enter. The same analysis found that Dirichlet scoring, like BM25 and pivoted TF-IDF, satisfies some basic retrieval constraints only for certain parameter ranges.

When it breaks

Unigram independence ignores word order. "Hot dog" and "dog hot" score identically, and nothing rewards query terms appearing near each other. Proximity and sequential-dependence features have to be added outside the model.

Scores grow with query length. The log-likelihood is a sum over query terms and the length penalty scales with \(|q|\), so scores are not comparable across queries and thresholding them is unsound, just as with BM25.

Unseen collection terms are a real zero. A query term absent from the corpus has \(P(t \mid C) = 0\), so implementations floor it, and different floors rank rare-term queries differently.

Verbose queries punish Dirichlet. A pasted paragraph or an LLM-generated query behaves like the long queries in the original experiments, where Dirichlet smoothing lost ground and heavier interpolation won. Retuning \(\mu\) for a keyword-query workload and then serving conversational queries silently moves the operating point.

The collection model inherits the corpus's noise. Boilerplate, navigation text and near-duplicate spam inflate \(P(t \mid C)\) for the terms they contain, which lowers the value of matching those terms everywhere.