Two-Stage Retrieval
Retrieving a broad candidate set cheaply and then reordering it with an expensive, more accurate model.
Embedding similarity is fast because query and document are encoded independently — the document vectors are precomputed, and the query is one comparison against an index. That independence is also why it is imprecise: the model never sees the query and the passage together.
A cross-encoder processes the pair jointly and scores relevance directly. Far more accurate, and far too slow to run against a whole corpus.
Two-stage retrieval uses each for what it is good at: retrieve 50–100 candidates by embedding similarity, rerank them with a cross-encoder, keep the top 5–10 for the prompt.
The gain is usually substantial, and it is the highest-return single addition to a naive retrieval pipeline — frequently larger than changing the embedding model or the vector store.
The costs are honest: added latency of tens to hundreds of milliseconds depending on candidate count and model, and an additional model to serve or an additional API dependency.
Two refinements: hybrid retrieval feeding the reranker from both vector and keyword search, which covers the exact-term weakness of embeddings; and reciprocal rank fusion for combining multiple ranked lists before reranking, which is simple and works well.