RAG Evaluation and Groundedness

A RAG system gives a wrong answer. There are exactly three places the fault can lie: the retriever never surfaced the right passage, it surfaced it and the generator ignored it, or the generator used it and reasoned badly. Teams that evaluate only the final answer cannot tell these apart, so they tune whichever component they happen to believe in. The first job of RAG evaluation is attribution, not scoring.

Two stages, two metric families

Retrieval metrics need a set of queries with known relevant documents. The standard three:

• Recall@k — fraction of relevant documents present in the top \(k\). This is the ceiling on everything downstream. If recall@50 is 0.7, no reranker or prompt makes the missing 30 percent appear.
• MRR — mean reciprocal rank of the first relevant document, sensitive to whether the answer is first or fifth.
• nDCG@k — discounted cumulative gain, the right choice when relevance is graded rather than binary.

Generation metrics need the generated answer and the retrieved context:

• Faithfulness / groundedness — are the claims in the answer supported by the retrieved context? Decompose the answer into atomic claims, check each against the context, report the supported fraction.
• Answer relevance — does the answer address the question that was asked?
• Context precision and recall — how much of the retrieved context was actually used, and did the context contain what was needed?

The RAGAS framework put these on a common footing and, importantly, made them reference-free, so they can run without a human-written gold answer for every query (Es et al., 2023, RAGAS, arXiv:2309.15217). ARES trains lightweight judges with confidence intervals over the same dimensions (Saad-Falcon et al., 2023, arXiv:2311.09476).

Groundedness is the tractable one

"Was this answer good?" is contested, subjective and expensive to label. "Is every claim in this answer traceable to a retrieved passage?" is close to a decidable question, and it catches the failure mode that matters most in enterprise deployments: a confident sentence that appears in no source. This is why groundedness has become the workhorse metric. It is checkable by a smaller model, it produces a per-claim audit trail rather than a single number, and it maps directly onto citation UX.

The limit is equally clear. Groundedness measures the answer against the retrieved context, not against the world. Retrieve an outdated policy document and a perfectly grounded answer is still wrong. Groundedness bounds hallucination; it does not bound error.

Building the eval set

Human-written query sets are the gold standard and few teams sustain them. The practical path is synthetic generation with human filtering: sample chunks from the corpus, have a model write questions each chunk answers, keep the chunk as the labelled positive, then have a human discard the third of them that are trivially keyword-matchable or badly formed. A hundred well-curated queries covering your real query mix beat a thousand generated ones, mostly because a synthetic set generated from single chunks contains no multi-hop, no negation, and no "not in the corpus" cases. Add those by hand.

Two things belong in every set: unanswerable queries, where the correct behaviour is refusal, and queries with near-miss distractors, where a plausible wrong passage sits close to the right one. Both are where production systems actually fail, and neither appears in synthetic data by default.

When it breaks

• LLM judges are biased. They prefer longer answers, answers in their own style, and their own generations. Use a different model family for judging than for generating, and validate the judge against human labels on a sample before trusting it. See custom evals and LLM judges.
• Averaged scores hide the tail. A mean faithfulness of 0.92 is compatible with 8 percent of answers being confidently fabricated. Report the distribution and inspect the bottom decile.
• The eval set ages. It is built from a corpus snapshot. When the corpus changes, labels silently rot, and a "regression" is really a stale label.
• Component metrics do not compose. Recall@50 up and faithfulness up does not guarantee end-to-end quality up, because retrieving more can push the right passage into the lost-in-the-middle region. Always keep one end-to-end number alongside the component metrics.