BM25 Refuses to Die: The Probabilistic Relevance Framework Behind Search's Most Durable Baseline

In April 2021 Nandan Thakur and colleagues published BEIR, 18 retrieval datasets chosen to test whether retrievers trained on MS MARCO work anywhere else. Dense Passage Retrieval, which a year earlier had beaten BM25 by 9 to 19 points of top-20 accuracy on open-domain QA (Karpukhin et al., 2020, arXiv:2004.04906), scored 47.7% below BM25 on average. The best dense model in the study, TAS-B, still trailed by 2.8%. The abstract put it plainly: "BM25 is a robust baseline" (Thakur et al., 2021, BEIR, arXiv:2104.08663).

The function they lost to has two parameters and was submitted to TREC in 1994 by the Okapi group at City University London. BM25 ends a thirty-year derivation that starts with a 1960 paper on library indexing, passes through a theorem about optimal ranking, and finishes with a curve fitted by hand to a mixture of two Poisson distributions. Each factor exists because a modelling assumption demanded it, and each production failure traces to an assumption that did not hold.

Why this matters: BM25 is the default scorer in Lucene-based engines, the lexical arm of most hybrid RAG stacks, and the baseline every retriever is measured against. The Probabilistic Relevance Framework tells you what \(k_1\) and \(b\) model, why scores cannot be thresholded, and which failures are tunable and which are structural.

TL;DR

• BM25 is derived: rank by probability of relevance (the PRP, 1977), assume term independence given relevance, add a hidden "eliteness" variable with Poisson term counts, then replace the intractable result with a fitted saturation curve.
• IDF is the 1976 Robertson/Sparck Jones relevance weight with \(R = r = 0\). If 8 of 10 judged-relevant documents contain a term found in 20 of 1,000 documents, its weight rises from 3.87 to 5.58.
• Saturation follows from the model: the exact 2-Poisson weight approaches a ceiling as term frequency grows, and \(tf/(k_1 + tf)\) imitates that shape with one parameter.
• \(b\) encodes a theory of why documents are long. In the worked example, moving \(b\) from 0.75 to 0 flips which of two documents ranks first, and Lucene's and Anserini's defaults disagree about the same pair.
• The variant barely matters: eight BM25 variants on Robust04 span average precision 0.2516 to 0.2571 with no significant differences (Kamphuis et al., 2020).
• BM25 averages 0.423 nDCG@10 across BEIR's 18 datasets (computed from the paper's table). BM25 plus a cross-encoder beats it on 16 of 18; TAS-B on only 8.
• Jimmy Lin called neural ranking gains "hype" against weak baselines in 2018 and recanted in 2019 once pretrained transformers arrived. BERT took the reranking stage; BM25 kept the first stage.

At a Glance

Purple steps are algebra or explicit modelling assumptions; amber steps are where theory stopped yielding closed forms and the Okapi team fitted curves.

flowchart LR
    A["Probability Ranking Principle"] --> B["Odds and Bayes inversion"]
    B --> C["Term independence given relevance"]
    C --> D["RSJ weight, IDF with no judgments"]
    D --> E["Eliteness, 2-Poisson mixture"]
    E --> F["Saturating tf approximation"]
    F --> G["Soft length normalisation"]
    G --> H["BM25"]
    H --> I["BM25F across fields"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class A blue
    class B,C,D,E purple
    class F,G amber
    class H,I teal

From Library Catalogues to TREC

Maron and Kuhns proposed that an indexer assigning a subject heading should estimate the probability that a user looking under it would find the book relevant, and that the system should rank by those probabilities (Maron and Kuhns, 1960, JACM 7(3)). What was missing was a way to estimate the probabilities.

Karen Sparck Jones supplied the statistics. In 1972 she showed that weighting terms by how few documents contain them, "term specificity", now IDF, improved retrieval across collections (Sparck Jones, 1972, Journal of Documentation 28(1)). The justification was empirical, and the theory stayed contested long enough that Robertson was still publishing on theoretical arguments for IDF in 2004 (Robertson, 2004).

Robertson and Sparck Jones joined the two in 1976, deriving term weights from counts of relevant and non-relevant documents containing each term (Robertson and Sparck Jones, 1976, JASIS 27(3)). A year later Robertson stated the Probability Ranking Principle: ranking by decreasing probability of relevance gives the best effectiveness obtainable from the data, under stated conditions (Robertson, 1977, Journal of Documentation 33(4)).

That model still ignored how often a term occurs. Separately, Bookstein and Swanson (1974) and Harter (1975, JASIS 26(4)) had found that counts of a content-bearing word across abstracts fit a mixture of two Poisson distributions: documents that merely mention the word, and documents about it. That second population became eliteness.

timeline
    title The Probabilistic Relevance Framework
    1960 : Maron and Kuhns rank by probability of relevance
    1972 : Sparck Jones introduces term specificity, later IDF
    1975 : Harter fits a 2-Poisson mixture to term counts
    1976 : Robertson and Sparck Jones derive the RSJ weight
         : The PRP follows in 1977
    1993 : Okapi BM11 and BM15 at TREC-2
    1994 : BM25 at TREC-3 and SIGIR
    2004 : BM25F combines fields before saturation
    2016 : Lucene 6 makes BM25 the default similarity
    2021 : BEIR finds BM25 a robust zero-shot baseline

Theory became engineering at TREC. At TREC-2 the Okapi group submitted "Best Match" functions: BM1 used the RSJ weight alone, BM15 added a 2-Poisson-motivated term frequency factor, and BM11 added full length normalisation. Average precision on their ad hoc runs went 0.199, 0.227, 0.300 (Robertson et al., Okapi at TREC-2). BM25 added a parameter \(b\) interpolating between BM15 and BM11, was run at TREC-3, and its derivation appeared at SIGIR the same year (Robertson and Walker, 1994).

Industry lagged two decades: Lucene kept TF-IDF as default until LUCENE-6789, released in Lucene 6.0, made BM25 the default (Apache Lucene, LUCENE-6789). Elasticsearch 5.0 followed.

[IMAGE: Two panels. Left: a library catalogue card annotated "P(user under this heading finds this book relevant) = 0.7", labelled Maron and Kuhns 1960. Right: a bar chart of Okapi TREC-2 average precision for BM1 (0.199), BM15 (0.227) and BM11 (0.300), with arrows labelled "add tf" and "add length normalisation". Caption: "From a librarian's guess to measured gains: length normalisation was the largest single step."]

How the Framework Builds BM25

The canonical account is Robertson and Zaragoza's monograph, which names the family the Probabilistic Relevance Framework (PRF) to separate it from language-model and divergence-from-randomness approaches (Robertson and Zaragoza, 2009, FnTIR 3(4)). The derivation below follows it.

From probability of relevance to a sum of weights

Take binary relevance \(\mathrm{Rel}\) for query \(q\) and a document \(d\) represented by term frequencies. Any strictly monotone transformation preserves ranking, written \(\propto_q\). Convert the probability to odds, apply Bayes' rule, and drop the prior odds, which do not depend on the document:

Two assumptions hide in the approximation: terms are independent conditional on relevance (weaker than it sounds; Cooper showed "linked dependence" suffices), and non-query terms are uncorrelated with relevance. Take logs, subtract each query term's zero-frequency weight (a per-query constant), and the score becomes a sum over query terms present in the document:

This "removing the zeros" step is why BM25 runs on inverted indexes: documents sharing no query term are never touched. Note what was discarded. The dropped constants mean the score ranks like a probability but is not one, and Robertson and Zaragoza state the transformations cannot be reversed into calibrated estimates.

The RSJ weight, and IDF as its empty case

Make term frequency binary. With \(N\) documents, \(n_i\) containing term \(i\), \(R\) judged relevant and \(r_i\) of those containing the term, estimating the four probabilities with a 0.5 pseudo-count gives the Robertson/Sparck Jones weight:

With no judgments, set \(R = r_i = 0\). The two 0.5 factors cancel:

Sparck Jones's heuristic reappears as a relevance weight, under the assumptions that a query term is as likely present as absent in relevant documents and that the collection stands in for the non-relevant set. IDF is the best guess before feedback; pseudo-relevance feedback replaces that guess with counts from top-ranked documents.

Eliteness and the 2-Poisson model

Term frequency carries evidence indirectly. Twelve occurrences of "Poisson" suggest the document is about Poisson processes, not that it is twice as relevant as one with six. The PRF introduces a hidden binary property of each document-term pair, eliteness, glossed as aboutness. Relevance affects eliteness; eliteness affects term frequency; given eliteness, frequency is independent of relevance. Following Harter, frequencies are Poisson with mean \(\lambda_1\) in elite documents and \(\lambda_0 < \lambda_1\) otherwise. With \(p_1 = P(\mathrm{elite}\mid\mathrm{rel})\), \(p_0 = P(\mathrm{elite}\mid\overline{\mathrm{rel}})\) and \(E_e(tf)\) the Poisson probability under class \(e\):

This weight is zero at \(tf = 0\), rises monotonically, and approaches a finite ceiling. When elite documents rarely have zero occurrences, the ceiling is approximately

the weight eliteness itself would get if it were observable. The most a term can tell you is that the document is about it. That is saturation, and it shows why linear TF-IDF is wrong: the ceiling is infinite only if eliteness coincides exactly with relevance (\(p_1 = 1\), \(p_0 = 0\)).

Fitting a curve instead of a mixture

The exact weight needs \(p_1\), \(p_0\), \(\lambda_1\) and \(\lambda_0\) per term, with no easy way to estimate them; the Okapi TREC-2 paper calls the exact formula intractable in practice. So the Okapi team imitated its shape with one parameter, using the RSJ weight as the ceiling:

Small \(k_1\) saturates fast (\(k_1 = 0\) is binary presence); large \(k_1\) stays near linear longer. The monograph notes that real 2-Poisson curves are mostly convex but sometimes start concave. A hyperbola cannot do that, and the worked example shows the cost.

Verbosity versus scope

Harter's model assumed fixed-length abstracts. The PRF offers two reasons documents are long. Verbosity: the same content in more words, so divide counts by length. Scope: more topics, like concatenated documents, so do not. Collections mix both, so BM25 normalises softly, dividing \(tf\) by \(B = (1 - b) + b \cdot dl/avdl\) before saturation. With the conventional \((k_1+1)\) numerator, which changes no ranking and which Lucene 8 omits:

\(b = 1\) is BM11 (pure verbosity), \(b = 0\) is BM15 (pure scope). The theory sets neither \(k_1\) nor \(b\). Robertson and Zaragoza report \(0.5 < b < 0.8\) and \(1.2 < k_1 < 2\) as "reasonably good in many circumstances", with optima depending on the collection.

Fields: combine, then saturate

Summing per-field BM25 scores assumes a separate, independent eliteness per field, as if a document could be about a term in its title but not its body. BM25F treats eliteness as document-level and merges field evidence before saturating (Robertson, Zaragoza and Taylor, 2004, CIKM):

with field weight \(v_s\), field length \(sl_s\) and per-field \(b_s\). The monograph reports tuned values of \(k_1 = 27.5\) and title weight 38.4 for TREC 2003 topic distillation. Elasticsearch's combined_fields query approximates the simple variant (Elastic docs); summing per-field match scores double counts one piece of evidence.

Seeing It in Motion

The eliteness model as structure. Relevance never touches term frequency directly, and the rose node is the formula nobody could estimate.

flowchart TB
    subgraph Model["Generative model for one query term"]
        R["Relevance to query"] --> E["Eliteness, hidden"]
        E --> P1["Poisson rate lambda1 if elite"]
        E --> P0["Poisson rate lambda0 if not"]
        P1 --> TF["Observed term frequency"]
        P0 --> TF
    end
    TF --> X["Exact weight, four unknowns"]
    X --> S["Fit tf over k1 plus tf"]
    L["Length relative to average"] --> S
    I["IDF as the ceiling"] --> W["BM25 term weight"]
    S --> W

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class R,TF,L blue
    class E,P1,P0 purple
    class X rose
    class S,I amber
    class W teal

What a production engine actually does, which is where several failures below originate: the analyzer defines terms, the index stores a lossy length, and IDF comes from the shard.

sequenceDiagram
    participant App as Application
    participant An as Analyzer
    participant Ix as Shard index
    participant Sc as BM25 scorer
    participant Hp as Top-k heap
    App->>An: raw query text
    An->>Ix: normalised, stemmed terms
    Ix->>Sc: docFreq and docCount for this shard
    Ix->>Sc: postings with tf and 1-byte length norm
    Note over Sc: idf times saturated tf per term
    Sc->>Hp: candidate score
    Hp-->>Sc: threshold for skipping blocks
    Hp->>App: top-k ids and raw scores

[IMAGE: Line chart, term frequency 0 to 12 on x, term weight on y. TF-IDF as a straight rising line labelled "unbounded"; BM25 curves for k1 = 0.5, 1.2 and 3.0 flattening toward a dashed IDF ceiling; the exact 2-Poisson curve from the worked example drawn as an S-shape in a contrasting colour. Caption: "k1 sets how fast repetition stops mattering; the true 2-Poisson curve starts slower than any hyperbola."]

By the Numbers

Sources: Okapi at TREC-2; Kamphuis et al., 2020; Anserini MS MARCO passage docs; BEIR, where relative figures are the paper's and the 0.423 mean and win counts are computed here from its per-dataset table.

The TREC-2 jump from BM15 to BM11, 0.073 AP, is thirteen times the spread between modern variants: length normalisation was the big idea. Its largest deficits include question-style sets such as NQ (BM25 0.329, TAS-B 0.463).

A Concrete Example

\(N = 1{,}000\) documents, \(avdl = 100\) tokens, query poisson retrieval. "poisson" appears in 20 documents, "retrieval" in 300.

• Doc A, 60 tokens: "poisson" once, "retrieval" six times.
• Doc B, 240 tokens: "poisson" three times, "retrieval" twice.

Step 1: IDF. \(\ln(980.5/20.5) = \ln 47.83 = 3.868\) for "poisson"; \(\ln(700.5/300.5) = \ln 2.331 = 0.846\) for "retrieval".

Step 2: length factor at Lucene defaults (\(k_1 = 1.2\), \(b = 0.75\)). Doc A: \(B = 0.25 + 0.75 \times 0.6 = 0.70\), so \(k_1 B = 0.84\). Doc B: \(B = 0.25 + 0.75 \times 2.4 = 2.05\), so \(k_1 B = 2.46\). B's effective \(k_1\) is nearly three times A's.

Step 3: saturate, \(tf(k_1+1)/(tf + k_1 B)\), and weight.

Step 4: sum. A \(= 6.258\), B \(= 5.510\). A ranks first. B's three "poisson" occurrences earn 1.209 saturated units against A's one at 1.196: the model attributes them mostly to B's length.

Step 5: change the theory of length. At \(k_1 = 1.2\) the winner flips at \(b = 0.51\).

Two widely used defaults rank this pair differently, and at \(k_1 = 0\) the documents tie, so every difference comes from frequency and length. Neither engine is wrong; only judgments on your own collection settle verbosity versus scope.

Step 6: what the hyperbola misses. Take a term with \(p_1 = 0.5\), \(p_0 = 0.05\), \(\lambda_1 = 2.0\), \(\lambda_0 = 0.3\). The exact 2-Poisson weight at \(tf = 1, 2, 3, 8\) is \(0.58, 1.70, 2.50, 2.79\). The BM25 shape with \(k_1 = 1.2\), scaled to the ceiling \(\ln 19 = 2.94\), gives \(1.34, 1.84, 2.10, 2.56\). Stray mentions in non-elite documents make one occurrence weak evidence; BM25 over-credits it. The plateau sits below \(\ln 19\) because \(E_1(0) = e^{-2}\) is not negligible.

Step 7: add feedback. A user marks \(R = 10\) documents relevant; \(r = 8\) contain "poisson":

The weight rises 44% over IDF's 3.868, from one extra column of counts.

[IMAGE: Worked-example panel. Top: Doc A and Doc B as horizontal bars proportional to 60 and 240 tokens, orange ticks for "poisson", blue for "retrieval". Middle: per-term contribution bars at b = 0.75. Bottom: total score against b from 0 to 1 for each document at k1 = 1.2, crossing at b = 0.51, with b = 0.75 and b = 0.4 marked. Caption: "Which document wins depends on which theory of document length you believe."]

Where It Breaks

Vocabulary mismatch is structural

The PRF sums over query terms only. A document about "automobiles" scores zero for "car" at every \(k_1\) and \(b\), the usual explanation for its BEIR losses on question-style data. Query expansion helps but, as the monograph notes, adds correlated synonyms that strain the independence assumption. Learned sparse models such as SPLADE predict expansion terms and weights while keeping the inverted index (Formal et al., 2021, arXiv:2107.05720).

Very long documents are over-penalised

For a document ten times average length at \(b = 0.75\), \(B = 7.75\), so one occurrence saturates to about \(2.2/10.3 = 0.21\) against 1.0 in an average document. Lv and Zhai showed the normalisation is not lower-bounded, so a very long matching document can score like a non-match; their title is "When documents are very long, BM25 fails!" (Lv and Zhai, 2011, SIGIR). BM25+ and BM25L add a constant \(\delta\). Chunking into passages sidesteps it.

[IMAGE: Two-panel plot. Left: saturated tf contribution of a single occurrence against document length from 0.5x to 20x average, for b = 0.25, 0.75 and 1.0, with a horizontal dotted line at BM25+'s delta floor. Right: Robertson IDF versus Lucene IDF against document frequency from 0 to 100% of the collection, shading the region where Robertson's goes negative. Caption: "Two quiet failure modes: long documents fade toward zero, and common terms go negative."]

IDF goes negative, and engines disagree on the fix

Robertson IDF is negative when \(n_i > N/2\): a term in 600 of 1,000 documents gets \(\ln(400.5/600.5) = -0.405\), so containing it lowers the score. Lucene uses \(\ln(1 + (N - n + 0.5)/(n + 0.5))\), giving 0.511; ATIRE uses \(\ln(N/n)\). Kamphuis et al. found no significant effect on newswire, but on homogeneous catalogues where "shirt" is in most documents the sign matters, and raw scores stop being comparable across engines.

Scores are ranks in disguise

The derivation dropped every document-independent constant, so a score of 14.2 means nothing outside its query. Thresholding BM25 to decide "nothing relevant found", or linearly mixing it with cosine similarity, treats a rank-equivalent quantity as calibrated. Use rank fusion or per-query normalisation.

IDF belongs to the shard

Elasticsearch's default query_then_fetch computes document frequencies per shard, so small indexes over many shards rank inconsistently; dfs_query_then_fetch gathers global statistics at extra latency (Elastic, Practical BM25, Part 1).

The analyzer decides more than the formula

Lucene's one-byte lossy length norm was long blamed for weak baselines, yet Kamphuis et al. measured 0.2531 AP against 0.2533 exact, and noted stopword choices often matter more than the variant. When two BM25 systems disagree, check tokenisation first.

Weak baselines, argued in public

Armstrong et al. found ad hoc retrieval papers reporting gains over baselines weaker than results already published (Armstrong et al., 2009, CIKM). Lin's 2018 essay doubted neural rankers beat tuned bag-of-words expansion without vast training data (Lin, 2018, SIGIR Forum 52(2)); a Robust04 meta-analysis found the best reported results were a decade old (Yang et al., 2019, arXiv:1904.09171). A year later, in "The Neural Hype, Justified! A Recantation", Lin wrote that pretrained transformers are "unequivocally more effective" even in the low-resource regime, and that his error was assuming effective rankers needed vast relevance judgments (Lin, 2019, SIGIR Forum 53(2)).

Alternative Designs

Query likelihood is a cousin: the monograph notes eliteness is itself a simple generative topic model. Dense approaches are complements: Wang et al. claimed in late 2022 that E5 was "the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data" (Wang et al., 2022, arXiv:2212.03533). That this merited a sentence in an abstract measures the baseline.

[IMAGE: Quadrant chart, x-axis "training data required", y-axis "robust to vocabulary mismatch". BM25 and query likelihood bottom-left, docT5query and SPLADE upper-middle, dense and cross-encoders upper-right, with an arrow from BM25 to the cross-encoder labelled "supplies candidates". Caption: "Every alternative buys vocabulary robustness with training data. BM25 is the point that needs none."]

How It Is Used in Practice

Default search engines. Lucene's BM25Similarity ships \(k_1 = 1.2\), \(b = 0.75\), inherited by Elasticsearch, OpenSearch and Solr and usually left untuned; Anserini uses \(k_1 = 0.9\), \(b = 0.4\). Since defaults disagree, tuning both on a few hundred judged queries is among the cheapest relevance gains available.

First stage of multi-stage ranking. Kamphuis et al. note neural rankers are "typically deployed as part of a multi-stage reranking architecture" over BM25-style candidates. Bounded per-term contributions let WAND skip most postings while returning the exact top-k.

Lexical arm of hybrid RAG. A 2024 study of Touché-2020 found neural retrievers returned short, non-argumentative passages; after removing passages under 20 words and adding judgments, neural nDCG@10 rose by up to 0.52 and BM25 was still more effective (Thakur et al., 2024, SIGIR, arXiv:2407.07790). For error codes, SKUs and identifiers, the lexical arm is often the only one that works.

[IMAGE: Hybrid search stack. Query splits into a BM25 inverted-index arm (inset showing k1, b, field weights) and a dense ANN arm; both return top-1000; rank fusion merges; a cross-encoder reranks top-100; an LLM reads top-10. Latency budgets annotated per stage. Caption: "BM25 rarely produces the final ranking, and almost always produces the candidates for it."]

Insights Worth Remembering

1. BM25 is a model whose estimation step was replaced by curve fitting. Eliteness defines the right weight; \(tf/(k_1 + tf)\) copies its shape. The robustness comes from the shape, not the constants.

2. IDF is relevance feedback with no feedback. RSJ with \(R = r = 0\) yields IDF, and every expansion method is a way of filling the empty column.

3. Saturation is what aboutness looks like in counts. It is also the per-term bound that makes dynamic pruning exact.

4. \(b\) is a stance on why documents are long. Collections mix verbosity and scope in proportions no default knows, which is why respected defaults disagree.

5. BM25 scores are not probabilities. Thresholds and linear score mixing inherit an error made at step one of the derivation.

6. Out of domain, having no training is an advantage. BM25 has no learned prior to be wrong about; DPR's 47.7% BEIR deficit is the price of one.

7. The neural camp beat BM25 by building on it. The best 2021 configuration was BM25 plus a cross-encoder, ahead on 16 of 18 datasets.

Open Questions

What should IDF mean across fields and shards? Robertson and Zaragoza computed IDF over whole documents regardless of field and flagged degenerate cases with verbose fields as needing "further research". Global statistics fix shard skew at a latency cost; a principled field-aware IDF that consistently helps has not been shown.

How much zero-shot edge survives modern embedding training? Measured: E5 and later embedders report beating BM25's BEIR average. Unclear: how much reflects training data overlapping benchmark domains.

Is the PRP the right objective when an LLM reads the results? The PRP judges each document independently and is optimal for an individual user under that assumption. A generator consuming ten passages values coverage and non-redundancy, which the PRP ignores. A PRF-style theory of set-level utility does not yet exist; diversity-aware reranking is the practical patch.

Should \(k_1\) vary by term? Lv and Zhai's adaptive BM25 derives term-specific \(k_1\), but Kamphuis et al. found the optimum undefined for about 90% of terms and no significant gain. Whether query-dependent parameters would help remains untested.

Sources and Further Reading

1. Maron, M. E., & Kuhns, J. L. (1960). "On Relevance, Probabilistic Indexing and Information Retrieval." JACM, 7(3), 216-244. doi:10.1145/321033.321035
2. Sparck Jones, K. (1972). "A Statistical Interpretation of Term Specificity and Its Application in Retrieval." Journal of Documentation, 28(1), 11-21. doi:10.1108/eb026526
3. Harter, S. P. (1975). "A Probabilistic Approach to Automatic Keyword Indexing. Part I." JASIS, 26(4), 197-206. doi:10.1002/asi.4630260402
4. Robertson, S. E., & Sparck Jones, K. (1976). "Relevance Weighting of Search Terms." JASIS, 27(3), 129-146. doi:10.1002/asi.4630270302
5. Robertson, S. E. (1977). "The Probability Ranking Principle in IR." Journal of Documentation, 33(4), 294-304. doi:10.1108/eb026647
6. Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M., & Gatford, M. "Okapi at TREC-2" and "Okapi at TREC-3." TREC-2 and TREC-3 Proceedings, NIST. TREC-2, TREC-3
7. Robertson, S. E., & Walker, S. (1994). "Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval." SIGIR '94, 232-241. doi:10.1007/978-1-4471-2099-5_24
8. Robertson, S., Zaragoza, H., & Taylor, M. (2004). "Simple BM25 Extension to Multiple Weighted Fields." CIKM 2004, 42-49. doi:10.1145/1031171.1031181
9. Robertson, S., & Zaragoza, H. (2009). "The Probabilistic Relevance Framework: BM25 and Beyond." FnTIR, 3(4), 333-389. doi:10.1561/1500000019
10. Armstrong, T. G., Moffat, A., Webber, W., & Zobel, J. (2009). "Improvements That Don't Add Up: Ad-Hoc Retrieval Results Since 1998." CIKM 2009, 601-610. doi:10.1145/1645953.1646031
11. Lv, Y., & Zhai, C. (2011). "When Documents Are Very Long, BM25 Fails!" SIGIR 2011, 1103-1104. doi:10.1145/2009916.2010070
12. Lin, J. (2018). "The Neural Hype and Comparisons Against Weak Baselines." SIGIR Forum, 52(2), 40-51. doi:10.1145/3308774.3308781
13. Lin, J. (2019). "The Neural Hype, Justified! A Recantation." SIGIR Forum, 53(2), 88-93. doi:10.1145/3458553.3458563
14. Yang, W., Lu, K., Yang, P., & Lin, J. (2019). "Critically Examining the 'Neural Hype'." SIGIR 2019. arXiv:1904.09171
15. Kamphuis, C., de Vries, A. P., Boytsov, L., & Lin, J. (2020). "Which BM25 Do You Mean? A Large-Scale Reproducibility Study of Scoring Variants." ECIR 2020, 28-34. doi:10.1007/978-3-030-45442-5_4
16. Thakur, N., Reimers, N., et al. (2021). "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models." NeurIPS 2021 Datasets and Benchmarks. arXiv:2104.08663
17. Thakur, N., Bonifacio, L., et al. (2024). "Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR." SIGIR 2024. arXiv:2407.07790
18. Apache Lucene. "LUCENE-6789: Change IndexSearcher Default Similarity to BM25." Jira

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.