Retrieval Is a Ranking Problem: Why Your RAG System Doesn't Need a Better Embedding Model

A team I talked to last year had a support-knowledge RAG system that got the right answer about two thirds of the time. Their fix was the obvious one: swap the embedding model. They moved from a mid-tier open model to whatever sat at the top of the MTEB leaderboard that month, re-embedded four million chunks over a weekend, redeployed, and measured. Answer quality moved by roughly nothing.

So they moved vector databases. Then they raised top-k from 5 to 20, which made things slightly worse. Then they raised the context window, which made things worse again. Six weeks in, someone finally read the retrieved chunks for twenty failing queries by hand and found that in fourteen of them the right passage had been retrieved and the model had answered from a different one, and in four of the remaining six the corpus contained the answer split across two chunks with the decisive sentence at a boundary.

None of that is an embedding problem. It is a ranking problem and an indexing-unit problem, in that order, and it is roughly the same problem that information retrieval spent thirty years on before anyone attached a generator to the end.

The three places an answer disappears

A RAG pipeline has exactly three failure surfaces, and they need to be measured separately, because a single end-to-end accuracy number cannot distinguish them.

graph LR
  Q[Query] --> A[Index<br/>recall ceiling]
  A --> B[Rank<br/>precision]
  B --> C[Generate<br/>groundedness]
  C --> D[Answer]

Recall failure. The passage is in the corpus but not in the candidate set. Nothing downstream can fix it. Every prompt improvement, every larger model, every reranker operates on candidates it was given.

Ranking failure. The passage is in the candidate set but sits at position 30, or at position 3 behind two plausible distractors that the generator liked better.

Generation failure. The passage is present, ranked first, and the model still answers from something else, or hedges, or fabricates around it.

The embedding model touches only the first of these, and only partially. Yet it absorbs most of the tuning effort, because it is the one component with a public leaderboard.

The leaderboard measures a different job than yours

MTEB spans eight task families across 58 datasets, and its own headline finding is that no method dominates: the field has not converged on a universal text embedding (Muennighoff et al., 2022, MTEB, arXiv:2210.07316). BEIR, which is the retrieval-specific ancestor, is blunter still. Across 18 datasets and 10 systems, it found that BM25 is a robust baseline, and that the architectures that actually win zero-shot are reranking and late-interaction models, at high computational cost (Thakur et al., 2021, BEIR, arXiv:2104.08663).

Read that finding carefully, because it is the whole argument of this post in one sentence. The zero-shot winners were not better bi-encoders. They were architectures that spend more compute per candidate at query time. A leaderboard position two points higher on an average over 58 heterogeneous datasets tells you very little about a corpus of Kubernetes runbooks with your company's internal service names in them, and the two points are, in any case, dwarfed by what a second ranking stage does.

There is also a mundane reason leaderboard gains evaporate. Most benchmark datasets have short, well-formed queries and passage-length documents. Production has a fourteen-word question, a chunk that is half a table, and an acronym that means something different inside your company than outside it. Domain shift eats the delta.

Two stages, because one stage cannot be both

The reason retrieval is architecturally a ranking problem is a hard constraint, not a preference. A bi-encoder embeds the document without ever seeing the query. That independence is exactly what lets you precompute four million vectors, and it is exactly what prevents the model from representing "this passage answers this question". It can only represent "this passage is about this topic".

A cross-encoder concatenates the query and the document and runs cross-attention over the pair, so every layer can relate a query term to a document term. It can handle negation, multi-clause conditions, and two near-identical passages that differ in one decisive number. It also cannot precompute anything, so it costs one forward pass per candidate and scales to tens of documents, not millions.

Neither is sufficient. The resolution the IR field reached long ago is to use both: a cheap, recall-oriented first stage over the whole corpus, then an expensive, precision-oriented second stage over a few dozen survivors. BERT rerankers took the MS MARCO leaderboard on this pattern in 2019 (Nogueira & Cho, arXiv:1901.04085), and the shape has not changed since; only the models inside it have.

The economics strongly favour a wide first stage. ANN search is sublinear in corpus size, so going from top-10 to top-100 costs almost nothing and materially raises the recall ceiling. Reranking is strictly linear in candidate count. So the design question is never "how many should I retrieve", it is "how many can I afford to rerank", and the answer is usually more than teams assume: a small cross-encoder over 50 passages batches well and lands in the tens of milliseconds on a GPU, against a generator that will spend seconds.

Lexical search is not legacy

The other half of the first stage is the one that people skip because it feels like a step backwards. BM25 is a bag-of-words scoring function from the 1990s, and it beats every dense retriever on a specific and very common class of query: the ones containing a token the embedding model has no useful representation for. Error codes. Part numbers. Internal service names. A customer's surname. Version strings. A dense retriever asked for ERR_2041 will return passages about errors in general, which is a semantically reasonable and operationally useless response.

Combining the two is a solved problem with an embarrassingly simple solution. Reciprocal rank fusion ignores scores entirely, which is the point, since scores from BM25 and cosine similarity are not on comparable scales, and sums over positions:

Cormack and colleagues published it in a two-page SIGIR paper in 2009 showing it beat Condorcet fusion and every individual learning-to-rank method they tested (Cormack, Clarke & Büttcher, SIGIR 2009, Reciprocal Rank Fusion). It has one parameter, needs no training, and remains the default hybrid fusion in production systems seventeen years later.

The unit you index outranks the model that embeds it

Here is where the ranking framing needs a caveat, because there is one decision upstream of ranking that dominates it: what a "document" is.

Chunking is usually treated as a preprocessing detail with a size parameter to tune. It is neither. It determines what can be retrieved at all, it cannot be changed without re-embedding the corpus, and its failures are silent. A chunk that ends mid-table produces a fragment of numbers with no header, which the reranker will happily score and the generator will confidently summarise. A chunk that begins with "such period may be extended by mutual agreement" has lost the antecedent for such period, and it will retrieve beautifully for queries about extensions while being incapable of answering them.

Two recent methods attack this directly and both are worth more than an embedding upgrade.

Contextual retrieval prepends an LLM-written situating sentence to each chunk before indexing. Anthropic reported that contextual embeddings combined with contextual BM25 cut top-20 retrieval failures by 49 percent, and by 67 percent with a reranker added (Anthropic, 2024, Contextual Retrieval). Note the shape of that result: the biggest single number in the study comes from stacking a lexical retriever and a reranker onto a better indexing unit. Three of the four ingredients are ranking-stack decisions.

Late chunking takes the opposite route. Run the entire document through a long-context embedding model, then pool token embeddings into chunk vectors after the transformer rather than before, so each chunk vector was computed with attention over the full document (Günther et al., 2024, arXiv:2409.04701). No extra LLM calls, and the pronouns resolve.

And for a large class of enterprise corpora, the most consequential development is that the text extraction step may not need to exist. ColPali embeds page images with a vision-language model and matches them with ColBERT-style late interaction, deleting OCR, layout detection, table extraction and reading-order reconstruction from the pipeline (Faysse et al., 2024, arXiv:2407.01449). If your documents are filings, slide decks and scanned reports, the parser has been quietly setting your recall ceiling all along.

Why more context is not a substitute

The recurring counterargument is that long context windows make this obsolete: stop ranking, stuff a million tokens in, let attention do retrieval. This does not work yet, for two independent reasons.

The first is measured. Model accuracy is not uniform across the input. Performance degrades when the relevant information sits in the middle of a long context, and the degradation is not gentle (Liu et al., 2023, Lost in the Middle, arXiv:2307.03172). Retrieving 100 passages instead of 5 can lower answer quality even when the extra 95 include the right one, because position now matters as much as presence. Ranking did not become unnecessary; it moved inside the prompt.

The second is economic. Prefill cost scales with prompt length on every request. A ranking stage that spends 40 ms to cut 100 candidates to 6 is not competing with a free alternative; it is competing with paying to prefill 90-odd irrelevant passages on every single call, forever.

The part nobody wants to do

If retrieval is a ranking problem, then it is an evaluation problem, because ranking quality is not observable by inspection. You need a query set with known relevant documents, and you need to measure the stages separately: recall@k for the index, nDCG or MRR after reranking, groundedness for the generator. RAGAS made the generation side of this tractable by defining reference-free metrics like faithfulness and answer relevance that do not need a gold answer for every query (Es et al., 2023, arXiv:2309.15217).

A hundred hand-curated queries covering your real query mix is enough to start and is worth more than a thousand generated ones. The catch with synthetic sets is structural: if you generate questions by asking a model to write a question each chunk answers, every query in your set is single-hop, answerable, and lexically close to its target. Production queries are multi-hop, sometimes unanswerable, and phrased in words that appear nowhere in the corpus. Add the unanswerable cases and the near-miss distractors by hand, because those are where deployed systems actually fail and where an eval set that lacks them will report a comfortable 0.94.

What this means on Monday

Rank the interventions by expected gain per unit of effort, and the usual priority list inverts:

1. Add a reranker. One dependency, tens of milliseconds, usually the largest single quality jump available.
2. Add lexical retrieval and fuse with RRF. Two-page paper, no training, fixes the entire class of exact-token queries that dense retrieval cannot see.
3. Fix the indexing unit. Respect document structure, add context to chunks, or drop the text extraction step entirely for visual corpora.
4. Build the eval set. A hundred real queries, split-stage metrics, unanswerable cases included.
5. Then, if the numbers say so, change the embedding model.

The embedding model is not unimportant. It is just the component with the highest ratio of attention received to quality delivered, largely because it is the one with a scoreboard. The stack that wins is the boring one that information retrieval converged on decades ago: cast a wide net cheaply, rank it expensively, and measure both halves separately.

Further reading

• Thakur et al., 2021, BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, arXiv:2104.08663
• Nogueira & Cho, 2019, Passage Re-ranking with BERT, arXiv:1901.04085
• Khattab & Zaharia, 2020, ColBERT, arXiv:2004.12832
• Anthropic, 2024, Introducing Contextual Retrieval
• Liu et al., 2023, Lost in the Middle, arXiv:2307.03172

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.