AI Executive Office — CXO Assistant Platform  ·  View 23 of 30  ·  6 · Operations

Observability

The signals, where they go, and the one metric that means the product is failing even when every service is green.

Editable source SVG draw.io All views
Emitted Turn trace OpenTelemetry GenAI Tool call log latency · failure Tokens and cost per tenant, per turn Answer quality grounded · abstained Platform metrics Azure resources Data freshness per source Collected Application Insights distributed trace Foundry tracing agent spans Azure Monitor metrics · diag Stored Log Analytics 90 d hot Archive 2 y · cheap tier Cost export to Fabric Watched SLO burn p95 answer latency Grounding regression rolling 24 h Tool failure rate per source Tenant spend budget per tenant Stale source past contract Acted on Page the SRE platform only Throttle a tenant protect the rest Show staleness in the answer Re-run evaluations before blaming data Reported Tenant health report monthly Adoption and value acceptance rate never silently Observability — Signals, and Who Gets Woken Up Application we own Security / platform Data store Decision point Interface / broker synchronous Recommendation acceptance is a platform metric, not a business one: if executives stop acting on what the platform surfaces, that is an outage of the product even when every service is green. Rejected answers feed the golden set in view 24. v 1.0 · owner Data & AI Global Practice · date 2026-09

What is measured

  • Latency, tokens and cost per turn per tenant; tool failure rate per source; grounding rate and abstention rate; data freshness per source against its contract
  • Recommendation acceptance is treated as a platform metric. If executives stop acting on what the platform surfaces, that is an outage of the product
  • Traces follow the OpenTelemetry GenAI conventions so a turn can be read end to end: prompt, plan, each tool call, model version, grounding verdict

The constraint

  • The platform SRE must diagnose a degraded tenant without reading tenant content. Telemetry carries identifiers, timings and verdicts — never prompts, retrieved passages or business values. Content-level traces stay inside the tenant boundary and are readable only by the tenant's own governance role
  • Stale data is shown in the answer, never handled silently

Numbers

  • 90 days hot in Log Analytics, 2 years archived
  • Cost per interaction is a tracked unit-economic metric with a per-tenant budget and an alert
  • SLO burn alerts on p95 answer latency and on grounding rate over a rolling 24 hours
◀  Delivery — Code and Configuration All views Model and Prompt Lifecycle  ▶
Solution Architecture v1.0 · Azure-native, multi-tenant, sovereign-capable · Data & AI Global Practice · 2026-0923-observability