Retrieval Pipeline Stages
The stages that turn a user question into grounded context — query processing, retrieval, reranking and assembly — each independently tunable.
Retrieval-augmented generation fails or succeeds in the retrieval half, and treating it as a single similarity search is the usual reason results disappoint.
Query processing. The user's question is often a poor search query — conversational, elliptical, or dependent on prior turns. Rewriting it, expanding it, or generating several sub-queries materially improves recall, and it is the cheapest improvement available.
Retrieval. Vector similarity finds semantically related passages and is weak on exact terms — product codes, names, error identifiers. Hybrid retrieval, combining vector search with keyword search, is the usual answer and is close to a default.
Reranking. Retrieve broadly (50–100 candidates), then reorder with a cross-encoder that scores the query and passage together. Slower per item and much more accurate than embedding similarity, which is why it belongs on a shortlist rather than on the whole corpus.
Assembly. Which passages fit in the context, in what order, with what metadata. Position matters — models attend unevenly across a long context — and provenance in the prompt is what makes citation possible.
The operational requirement that makes the whole thing improvable: an evaluation set measuring retrieval quality separately from generation quality. Without it, tuning is guesswork, because a bad answer may be a retrieval failure or a generation failure and they need different fixes.