Candidate Recall Ceiling

A team spends six weeks on a reranker and answer quality moves by two points. The reranker works; the problem is that on 22% of evaluation queries the correct passage was never in the candidate set, so no amount of reordering could put it first. Every euro spent downstream of the first stage was competing for the remaining 78%.

This number is the first thing to measure in any retrieval system and the last thing most teams measure. It is cheap: take 200 to 300 real queries, label the passage that should have been retrieved, and compute the fraction of queries where that passage appears in the first-stage top k. That single number tells you whether your problem is retrieval or ranking, and the two have entirely different fixes.

Why it matters

Retrieval quality debates are unfalsifiable without it. "The answers are wrong" is compatible with a bad chunker, a bad embedding model, a bad reranker, a bad prompt, or a model that ignores its context. The recall ceiling splits that into two questions that can be answered separately: was the right text available, and was it used well?

It also settles investment. If the ceiling at k=100 is 96%, a perfect reranker buys at most 4 points, and the engineering should go into generation or into abstention instead. If the ceiling is 70%, a reranker is close to worthless and the money belongs in hybrid search, chunking or query rewriting.

Implementation patterns

• Measure at several k. Recall at 10, 25, 100 and 500 on the same query set. The shape of that curve sets rerank depth: if recall at 25 is 0.94 and at 100 is 0.96, depth 100 costs four times the compute for two points.
• Label by passage id, not by answer text. An answer can be right for the wrong reason; a passage id is checkable and stable across model changes.
• Regenerate the label set when the chunker changes, because passage ids change with it. This is the maintenance cost, and it is the reason teams skip the metric.
• Run it in CI on every change to chunking, embedding model, index parameters or query preprocessing. It takes seconds once the set exists.
• Report it per query class. Navigational queries, long natural-language questions and keyword fragments have very different ceilings, and the aggregate hides which one is failing.

Industry example

The two-stage pattern this metric governs is the standard shape of web and enterprise search: a cheap high-recall first stage over the whole corpus, then an expensive model over a few dozen candidates. The public retrieval benchmarks that research groups have reported against since the late 2010s, such as the MS MARCO passage ranking dataset published in 2016, separate the two stages for exactly this reason, and published results are routinely given as both a first-stage recall figure and a final ranking figure, because a ranking score is uninterpretable without the recall it was measured on top of.

Failure scenarios

• Reranker credited with a fix it did not make. Recall was raised by a chunking change shipped the same week and nobody separated the two.
• Recall measured on synthetic queries generated from the documents themselves. Those queries quote the passage almost verbatim, so recall looks near-perfect and real user phrasing collapses it.
• Ceiling silently falls after an index rebuild with different parameters, and the only symptom is worse answers weeks later.
• A high ceiling with terrible answers, which correctly points the investigation at the generation prompt or the number of passages being passed in.

Trade-offs

The honest cost is human labelling. There is no way around it: a ceiling measured against labels a model produced inherits that model's blind spots.

When not to use it

If the corpus is small enough that every query can be scored exhaustively — a few thousand documents against a cross-encoder — there is no first stage and no ceiling. The same applies when retrieval is a database lookup by key rather than a search. Below roughly 10,000 documents, measure end-to-end answer quality and skip the stage decomposition, because the two stages are not separable in a system that has only one.

Interview question

Q: Your RAG assistant gives confidently wrong answers and the team proposes buying a commercial reranker. What one measurement would you insist on first, how would you construct it in a week, and what result would make you reject the proposal?

What a strong answer covers: labelling 200 to 300 real production queries with the passage that should have answered them; computing recall at the current first-stage k; rejecting the reranker if the ceiling is low, because reordering cannot retrieve what was never fetched; naming the alternative investments that raise the ceiling (hybrid lexical plus vector retrieval, better chunk boundaries, query rewriting); and noting that queries must come from real traffic rather than be generated from the documents.

Quick check

Quiz: First-stage recall at k=100 is 0.71 and your reranker is state of the art. What is the maximum end-to-end accuracy you can reach, and where should the next month go? — At most 71%, and the month goes into retrieval: hybrid search, chunking and query rewriting, not ranking.

Flashcard: Why can a perfect reranker still leave a RAG system at 70% accuracy? — Because reranking reorders the candidate set and never adds to it, so first-stage recall is a hard ceiling on everything downstream.