AI Agent Orchestration Platform  ·  View 31 of 32  ·  6 · Assurance and delivery

Failure Modes and Residual Risk

Eight ways this platform fails, what detects each, what contains it, and what is left over.

Editable source SVG draw.io All views
Signal Class Immediate response Containment Residual risk Model provider outage 5xx and timeout Provider Circuit break Fallback route Quality drift Rate limit exhaustion 429 burst Transient Backoff + jitter PTU spillover Latency spike Tool unavailable Health probe red Tool Retry then skip Degraded answer Incomplete result Agent loop divergence Step count rising Agent or model Depth and token cap Terminate run Wasted spend Poison message Delivery count Permanent Dead letter Triage and replay Manual backlog Policy or auth denial 403 on tool call Policy or auth Fail closed Escalate to human Workflow stalls Region loss Probe failure Infrastructure Front Door reroute Replay checkpoints RPO 5 min gap Control plane outage Registry 5xx Infrastructure Admissions stop Cached bundles run No new versions Failure Modes — Signal, Classification, Response and Residual Risk Security / platform Decision point Interface / broker Application we own Risk / gap Queue / topic Person or role Every row ends in a residual risk that is accepted, not eliminated. Those risks are the agenda for the operational review. v 1.0 · owner SRE and Architecture · date 2026-08

How to read this

  • Every row ends in a residual risk that is accepted rather than eliminated — those risks are the agenda for the operational review
  • Detection comes before response on every row; a failure mode with no signal is not managed, it is merely hoped about
  • Policy and authentication failures fail closed. Everything else degrades

Resilience mechanisms

  • Retries with exponential backoff and jitter, circuit breakers, timeouts, dead-letter queues, provider failover, workflow recovery, idempotency, backpressure, rate limiting and graceful degradation
  • Backpressure is applied at admission: when queue depth exceeds the class threshold, new low-priority runs are shed rather than queued indefinitely
  • Graceful degradation is explicit per capability — a run may complete with a documented tool omission rather than fail

Open items

  • Cosmos write-region failover is manual; the runbook exists but the automation does not, and the RTO depends on the on-call response
  • Dead-letter triage is partly manual; a recurring poison pattern needs an automated classifier before volume grows
  • The residual quality drift from provider fallback has no automated alarm yet — it is caught by the online evaluation, which lags
◀  Multi-Tenant Isolation All views Delivery Roadmap  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 32 views · Microsoft Azure31-failure-modes