A retrieval system's first-stage results are mediocre. Is reranking the right investment, and what does it cost?

Why reranking is usually the highest-return addition

First-stage retrieval optimises for recall at low cost across a large corpus: get the plausible candidates cheaply. It is deliberately approximate, and approximate methods rank imprecisely.

A reranker examines a small candidate set — tens of items — with a much more expensive model that considers the query and the candidate together rather than comparing precomputed representations. That interaction is what produces materially better ordering, and it is affordable precisely because the candidate set is small.

For most retrieval systems, adding a reranker improves result quality more than tuning the first stage does, because the first stage's job is coverage and the reranker's job is precision.

What it costs

• Latency. An additional model call on the critical path, roughly proportional to candidate count.
• Compute, scaling with candidates times query volume.
• A second model to operate, version and evaluate, with its own failure modes.
• A hard dependency unless it is designed to be optional.

How to make the trade well

• Rerank a small candidate set. The quality gain flattens well before the cost does; reranking a hundred candidates rarely beats reranking twenty-five by enough to justify it.
• Make it optional and fail open. If the reranker is slow or unavailable, serve first-stage results. This converts a hard dependency into a soft one, which is worth more than the last increment of quality.
• Cache aggressively for repeated queries, which in most workloads are a substantial fraction.
• Apply it only where it pays — a high-intent query where ordering matters, rather than every lookup.

What to fix first if the first stage is genuinely poor

Reranking cannot recover what retrieval never surfaced. If the correct answer is not in the candidate set, no reranker helps. So before reranking:

• Hybrid retrieval, since pure semantic search misses exact identifiers, codes and names.
• Metadata filtering to scope before ranking.
• Chunking that preserves structure, so relevant content is retrievable at all.

Diagnose which problem you have first: measure whether the correct source appears anywhere in the candidate set. If it usually does, rerank. If it usually does not, fix retrieval — reranking a set that does not contain the answer produces a better-ordered wrong result.