Retrieval-Generation Separation

A retrieval-augmented system that produces fluent, confident, wrong answers is the normal failure mode, and it is normally misdiagnosed. Fluency is not evidence that retrieval worked, so the output gives no signal about where the failure was.

The separation resolves it: for a set of questions with known answers, measure whether the correct passage was retrieved at all, and at what rank.

Why it matters

It divides an unfalsifiable problem into two tractable ones. If the correct passage was not retrieved, no amount of prompt engineering fixes it — and prompt engineering is where teams spend their effort by default. If it was retrieved and the answer is still wrong, the problem is in generation or in how context was assembled.

Most teams never make this measurement, and most RAG debugging is consequently guesswork.

Implementation patterns

• A golden set with known correct answers and known correct source passages. The second half is the part that is usually missing and is what makes the separation possible.
• Report retrieval metrics independently: was the correct passage in the top k, at what rank, and what proportion of retrieved passages were relevant.
• Run on every change to chunking, embedding model, retrieval strategy, reranking or prompts, because improvements in one area silently regress another and that is the default state without evaluation.

And the retrieval problems the measurement will surface, in rough order of frequency:

• Chunking that splits the answer. Follow document structure rather than a fixed token count, with overlapping windows to mitigate boundaries.
• Embedding mismatch between question and answer text, since pure semantic similarity retrieves topically related rather than answer-bearing passages. Hybrid keyword-plus-semantic retrieval reliably outperforms either alone.
• No reranking. Retrieving a wider candidate set and reranking with a cross-encoder is a large quality gain for modest cost, and is the most commonly omitted stage.
• Missing metadata filtering by tenant, category, language or date — which is a correctness and often a security issue rather than a quality one.
• A stale index.

Industry example

Marketplaces such as Meesho applying retrieval over catalogue, policy and support content encounter all of these, and the metadata-filtering case is the one with security consequences: retrieval that does not enforce tenancy or entitlement at query time can surface a seller's data to another seller.

Failure scenarios

• Only end-to-end evaluation, so failures cannot be localised.
• Golden set without source passages, making the separation impossible.
• Prompt engineering applied to a retrieval failure, which cannot work.
• Changing the vector store to improve quality, when quality is dominated by chunking, embeddings, hybrid search and reranking.
• No evaluation at all, so every change is a guess.

Trade-offs

Building and maintaining a golden set with source-passage annotations is real work, and it ages as the corpus and the usage change — requiring periodic refresh from production traffic.

The alternative is developing by intuition in a system where fluent output is uncorrelated with correctness, which is the specific property that makes intuition unreliable here. The evaluation cost is front-loaded and the alternative's cost is continuous and invisible.

Interview question

"Your RAG assistant gives a confidently wrong answer to a question your documentation clearly answers. Tell me the first thing you would check, and what you would conclude from each possible result."