Reranking

The two-stage design exists because of a computational trade. Embedding-based retrieval encodes documents independently and ahead of time, which is what makes searching millions of chunks fast, and that independence is exactly what limits its accuracy — the document was encoded without knowing the query.

A cross-encoder processes the query and candidate together, so it can judge relevance directly. That is far more accurate and far too slow to run across a whole corpus, so it runs across the top 50 or 100 candidates from the first stage and returns the best 5.

In practice this is often a larger quality gain than switching to a better embedding model, and it is a smaller change: no re-indexing, no migration, one component inserted into the pipeline.

The costs to plan for: additional latency in the region of tens to low hundreds of milliseconds depending on candidate count and model size, and additional inference cost per query. Both scale with how many candidates are reranked, which makes candidate count the main tuning knob.

The measurement that justifies it: evaluate retrieval separately from generation. Recall at 50 from the first stage tells you whether the right passage is even in the candidate set; precision at 5 after reranking tells you whether the model will see it. Teams that measure only end-to-end answer quality cannot tell which half to fix.