Enterprise Generative Search — Azure and Open Source  ·  View 33 of 41  ·  Operations

RAGOps Loop

How one bad answer becomes a fixed system and a permanent test.

Editable source SVG draw.io All views
Observe traces and feedback Classify the failure which stage broke Reproduce replay by evidence id Fix at the stage chunking, weights, prompt Evaluate offline against the golden set Ship behind a flag canary, then all Promote to golden the case becomes a test RAGOps weekly cadence a wrong answer, with its trace retrieval, rank, context or model the exact evidence set returns one change, one stage gate passes or the fix is wrong measured online for 48 hours it can never regress silently RAGOps — How a Bad Answer Becomes a Fixed System Security / platform Decision point Application we own Data store The loop closes only when the failing query joins the golden set. A fix that is not testable tomorrow is a patch, not an improvement. v 1.0 · owner Data and AI Global Practice

The decision

  • The loop closes only when the failing query joins the golden set. A fix that is not testable tomorrow is a patch, and the same failure returns within two releases.
  • Failures are classified by stage before anything is changed. Without the classification in view 34, teams argue about the model when the document was never retrieved.
  • One change per stage, per release. Two simultaneous fixes to retrieval and to the prompt produce a result nobody can attribute.

Numbers

  • Weekly cadence, about 40 triaged failures a week at current volume, of which roughly 60% are retrieval or corpus problems rather than generation ones.
  • The golden set grows about 30 queries a month from real failures, which is the main reason its difficulty tracks the product.

Assumptions

  • Traces are retained 90 days, which bounds how far back a reported failure can be reproduced. Failures older than that are reconstructed from provenance, which is kept for seven years.
◀  Evaluation Harness All views Observability and Tracing  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source33-ragops-loop