Enterprise Generative Search — Azure and Open Source  ·  View 32 of 41  ·  Operations

Evaluation Harness

Six layers of measurement, what each blocks, and how often each runs.

Editable source SVG draw.io All views
Golden set Measured by Gate Runs when Retrieval 620 queries, judged pools graded 0 to 3 Recall@50, nDCG@10, MRR trec_eval semantics Recall@50 at least 0.90 blocks release Every pull request fixture corpus Ranking Same pools, reranked pairwise preferences nDCG@10 delta vs fusion must be positive Gain at least 0.06 or the reranker is off Nightly and on model change full corpus Generation 480 queries with answers human-written references Groundedness, citation correctness Ragas plus NLI judge Groundedness at least 0.95 citations at least 0.97 Nightly, and every prompt change no exceptions End to end 300 task scenarios simple, multi-hop, long-tail Task success, abandonment and reformulation rate Task success at least 0.82 trend, not a single run Nightly, plus weekly online read from real traffic Safety and access 220 adversarial cases injection, leakage, PII Attack success rate and false refusal rate Zero access leaks a single failure blocks Every build, and on new sources never sampled Cost and latency Traffic-weighted replay yesterday's mix Cost per answered query, P95 per class Within 10% of baseline or approved explicitly Nightly reported with quality Evaluation Harness — What Is Measured, Against What, and What It Blocks Quality and cost are gated together. A change that improves groundedness by two points and doubles the bill is a decision, not an improvement. v 1.0 · owner Data and AI Global Practice

The decision

  • Evaluation is built before launch, not after. A generative search system with no golden set cannot tell an improvement from a regression, which means every change after that point is a guess.
  • Retrieval and generation are measured separately. An end-to-end score alone cannot say whether the answer was wrong because the document was never found or because the model ignored it.
  • Access safety has a zero-tolerance gate: one leak in the adversarial suite blocks the release, with no threshold and no waiver path.

Numbers

  • 1,400 golden queries: 620 retrieval with judged pools, 480 with human reference answers, 300 task scenarios, plus 220 adversarial cases.
  • Gates: Recall@50 at least 0.90, groundedness at least 0.95, citation correctness at least 0.97, task success at least 0.82, cost within 10% of baseline.
  • Composition is deliberate: 30% simple, 25% ambiguous, 20% multi-hop, 15% adversarial, 10% long-tail.

Assumptions and risks

  • Human reference answers are expensive and go stale as the corpus changes. Budget assumes two annotator-days a month to maintain the set.
  • An LLM judge is used for parts of the generation score and is itself validated against human labels quarterly. An unvalidated judge is a metric that drifts with a model release.
◀  CI/CD and Index Migration All views RAGOps Loop  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source32-evaluation-harness