A RAG system returns fluent answers that are frequently wrong. Where in the pipeline is the problem most likely, and how would you find out?

Where the problem usually is

Retrieval, not generation. A model given the right context usually produces a reasonable answer; a model given the wrong context produces a fluent, confident, wrong one. Fluency is not evidence that retrieval worked, which is what makes this failure mode so persistent — the output looks correct.

How to find out

Evaluate retrieval separately from generation, which most teams do not do. For a set of questions with known answers, measure whether the correct passage was retrieved at all, and at what rank.

If the correct passage is not in the retrieved set, no amount of prompt engineering fixes it. If it is present and the answer is still wrong, the problem is in generation or in how the context is assembled.

This single separation resolves most RAG debugging, and it turns an unfalsifiable "the model is bad" into a specific measurable defect.

The retrieval problems in rough order of frequency

• Chunking that splits the answer. A passage cut mid-explanation retrieves neither half usefully. Chunking should follow document structure rather than a fixed token count, and overlapping windows mitigate the boundary problem.
• Embedding mismatch between question and answer. Questions and documents are written differently, and pure semantic similarity retrieves passages that are topically related rather than answer-bearing. Hybrid retrieval — keyword and semantic combined — reliably outperforms either alone.
• No reranking. Retrieving twenty candidates and reranking with a cross-encoder is a large quality improvement for modest cost, and it is the most commonly omitted stage.
• Missing metadata filtering, so results include documents from the wrong category, tenant, language or time period. This is a correctness and often a security issue rather than a quality one.
• Stale index, where the answer exists and was indexed after the query, or was updated and not reindexed.

The evaluation infrastructure that must exist

A golden set of questions with known correct answers and known correct source passages, run on every change to chunking, embedding, retrieval or prompts. Without it, every change is a guess and improvements in one area silently regress another — which is the normal state of RAG systems that have no evaluation.