Enterprise Generative Search — Azure and Open Source  ·  View 34 of 41  ·  Operations

Observability and Tracing

One trace id across seven stages, so a wrong answer has a cause rather than an argument.

Editable source SVG draw.io All views
Span recorded Metric What it proves Alerts when Query understanding class, entities, rewrite and confidence class distribution vs the 7-day baseline The question was read correctly or was not class mix shifts 20% usually a UI change Authorisation group set size, fingerprint never the group names expansion latency P95 and cache hit rate The filter was applied and to what any query without a filter page immediately Retrieval per backend: query, depth, latency and every id returned recall proxy, zero-result rate per source The right document was found or never was zero-result rate above 4% per source Ranking fusion order, rerank scores before and after rank of the cited passage distribution It was found and then buried the commonest cause cited rank drifts past 6 ranking regression Context assembly slots, tokens, what was dropped and why evidence truncation rate The model was given enough or was starved truncation above 8% budget is wrong Model call model, tokens, cost, retries prompt version TTFT, total latency, spend It had what it needed and still failed a model problem TTFT P95 above 2.5 s or spend above budget Verification and answer claims, verdicts, citations abstentions recorded groundedness, abstain rate and feedback What the user was actually shown abstain rate above 6% corpus gap, not a bug Observability — One Trace That Answers Which Stage Was Wrong One trace id spans all seven rows. Without it, every quality question becomes an argument between the retrieval team and the model team. v 1.0 · owner Data and AI Global Practice

The question this view answers

  • Was the answer wrong because the document was never retrieved, because it was retrieved and buried, because the context was starved, or because the model synthesised badly? Each has a different owner and a different fix.
  • The rank of the cited passage is the single most diagnostic metric in the platform. When it drifts, ranking has regressed even if every other number looks normal.
  • Any query that reaches the index without an ACL filter pages immediately. It is the only alert in the set with no threshold.

Numbers

  • Traces store identifiers, not passage text; a trace is about 12 KB, and 90 days of traces is roughly 60 GB.
  • Alert thresholds: zero-result rate above 4% per source, evidence truncation above 8%, TTFT P95 above 2.5 s, abstain rate above 6%.
  • Sampling is 100% for traces and 100% for provenance; only prompt bodies are sampled, and only under an opt-in flag.

Risks

  • A rising abstain rate usually means a corpus gap rather than a platform fault, and is routed to stewards rather than to engineering. Misrouting it wastes the signal.
  • Storing prompts would make debugging easier and turn the trace store into a second copy of the corpus with different permissions. It is off by default and time-boxed when enabled.
◀  RAGOps Loop All views Cost Model  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source34-observability-and-tracing