Reranking and Cross-Encoders

Vector search computes the query embedding and the document embedding independently, then compares them. That independence is what makes it fast, because every document vector was computed offline, and it is also what makes it blunt: the document was encoded with no knowledge of the question. A passage about "reducing latency in Postgres" and a query about "speeding up slow Postgres queries" have to land near each other in a 1024-dimensional space on the strength of a general-purpose encoder's priors alone. Sometimes they do not.

A reranker fixes this by giving up the offline trick. It concatenates query and document, runs them through a transformer with full cross-attention, and emits a relevance score.

graph LR
  Q[Query] --> R[Retriever<br/>bi-encoder + BM25]
  R -->|top 50-100| X[Cross-encoder<br/>reranker]
  X -->|top 5-8| G[Generator]

Bi-encoder versus cross-encoder

The cross-encoder's every layer can attend from query tokens to document tokens, so it can represent "this document answers the question" rather than "this document is about the same topic". The idea predates LLM retrieval: BERT rerankers took the MS MARCO leaderboard in 2019 (Nogueira & Cho, arXiv:1901.04085), and monoT5 reframed the same task as sequence generation a year later (Nogueira et al., 2020, arXiv:2003.06713). Current production options are hosted rerankers (Cohere Rerank, Voyage), open cross-encoders (BGE-reranker, mxbai-rerank, Jina), and LLM-based listwise rankers such as RankGPT that score a whole candidate list in one prompt.

Sizing the two stages

The pipeline has one governing rule: the reranker cannot recover what the retriever never returned. Retrieval sets the recall ceiling; the reranker converts recall into precision.

That argues for a wide first stage. Retrieve 50 to 100 candidates (union of dense and lexical, fused with RRF), rerank them, keep the top 5 to 8. Going from k=10 to k=100 at the retriever costs almost nothing, since ANN search is sublinear, while reranking cost is strictly linear in k. So the tuning question is not "how many should I retrieve" but "how many can I afford to rerank".

A rough budget: a 300M-parameter cross-encoder scoring 50 passages of 500 tokens is 50 independent forward passes, which batches well on a GPU and lands in the tens of milliseconds, but a hosted reranker adds a network round trip and often 100 to 400 ms end to end. Against a generator that will spend seconds on the answer, that is usually the best latency a RAG system can spend.

Where it earns its keep

Rerankers help most exactly where embeddings are weakest: multi-clause queries where only part of the query matched, negation ("contracts without an arbitration clause"), near-duplicate passages differing in one decisive detail, and cross-lingual or jargon-heavy corpora. They also collapse the chunking problem somewhat, since you can retrieve generously and let the reranker discard.

When it breaks

• Long documents get truncated. Most cross-encoders cap around 512 tokens. A 2,000-token chunk is silently cut, and the reranker scores a fragment.
• Scores are not calibrated. A reranker score of 0.7 means nothing absolute and is not comparable across models or query types. Thresholding on it to decide "no good answer, refuse" needs per-corpus calibration; ranking with it is safe, filtering with it is not.
• Domain mismatch. A reranker trained on MS MARCO web passages can underperform on dense legal or clinical text. Measure on your own labelled pairs before assuming a gain.
• It is a second model in the hot path. Extra dependency, extra p99 tail, extra thing that can fail. Systems that cannot absorb the latency get more from a wider first stage and better chunking than from a reranker they must time out.