AI Agent Orchestration Platform  ·  View 27 of 32  ·  5 · Operations

Evaluation and Quality

The loop from production failure back to a better agent version, and the gate that closes it.

Editable source SVG draw.io All views
Capture production traces Curate golden datasets Evaluate Offline assertions + judge Gate promotion score Release Canary 5 percent traffic Evaluate Online live scoring Improve prompt · model · tool Agent Quality target 92 percent failed and escalated runs labelled cases scores per metric pass or block canary cohort live signal new candidate version Evaluation and Quality — The Loop That Closes Data store Application we own Decision point Security / platform Metrics tracked each turn: task success, groundedness, tool-call correctness, latency, tokens, cost, failure and human escalation rate. v 1.0 · owner Data & AI Global Practice · date 2026-08

Decisions

  • Failed and escalated production runs are the primary source of new evaluation cases; the golden set grows from reality, not from imagination
  • Offline evaluation gates promotion; online evaluation gates the canary. Both are required, and neither substitutes for the other
  • LLM-as-judge is used alongside deterministic assertions, never instead of them, and the judge itself is version-pinned

Metrics tracked

  • Task success rate, accuracy, groundedness, tool-call correctness, safety
  • Latency, token consumption, cost per successful task
  • Failure rate and human escalation rate — the two that tell you whether the agent is actually saving anyone time

Risks

  • A golden set drawn only from failures over-represents hard cases and can block a version that is better on average
  • Judge models drift between versions; the judge is pinned and its own agreement with human labels is re-measured each quarter
◀  Cost Management and Attribution All views Security Zones and Trust Boundaries  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 32 views · Microsoft Azure27-evaluation-quality-loop