Enterprise Generative Search — Azure and Open Source  ·  View 41 of 41  ·  Assurance

Failure Modes

Fifteen named failures across four planes, each with a detector and an owner.

Editable source SVG draw.io All views
Retrieval — the answer was wrong before the model saw anything How it fails Document never indexed parse failed silently Entitlement stale index behind the source Answer split across chunks neither is retrievable Embedding drift two model versions live Detected and answered by Per-document receipt steward console, view 08 60-second ACL delta plus cache invalidation Structure-aware chunking and coverage probes Alias cutover, never mixed view 31 Generation — the evidence was right and the answer was not How it fails Unsupported claim fluent and wrong Contradiction smoothed false consensus Over-refusal abstains on a good answer Schema break citations unparseable Detected and answered by Claim-level NLI drop or re-retrieve Contradiction detector both positions shown False refusal rate gate view 32 Validate and retry once then fall back Security — the boundary held, or it did not How it fails Cross-principal cache hit the highest-severity bug Injection reaches a tool document as instruction Prompt log holds content logging becomes a copy Detected and answered by Fingerprint in every key plus a nightly assertion Broker provenance check and egress allowlist Traces store ids only capture is opt-in Operations — it worked yesterday How it fails GPU pool exhausted reindex meets peak Source throttles the connector freshness silently slips Cost runaway one agentic loop pattern Model deprecated by the provider with notice, still disruptive Detected and answered by Separate pools and priorities batch is preemptible Freshness SLO per source staleness is published Budget governor and quota hard stop per query Gateway abstraction swap and re-evaluate Accepted risk — a correct answer from a stale but authorised document answered by publishing as-of, not by blocking Failure Modes — Named, Detected and Answered Every failure here is named, has a detector and has an owner. A failure mode with no detector is the one that becomes an incident. v 1.0 · owner Data and AI Global Practice

How to read this view

  • Every failure is paired with what detects it. A failure mode with no detector is the one that becomes an incident, because nothing distinguishes it from normal operation until a user complains.
  • The highest-severity defect in the platform is a cross-principal cache hit. It is guarded three ways — fingerprint in the key, nightly replay assertion, and a contract test that fails a build if a key is constructed without one.
  • The commonest real failure is not the model. It is a document that never indexed, or an entitlement that had not caught up.

Accepted risks

  • A correct answer drawn from a stale but authorised document. Answered by publishing an as-of rather than by withholding content.
  • Sixty seconds of entitlement lag on most sources; sources where that is unacceptable verify against the source before display.
  • Graph retrieval may not pay for itself. Its contribution is measured separately so removing it is a decision rather than an argument.

What has no mitigation

  • Entra ID unavailability is a full outage. No cached-entitlement fallback exists, deliberately.
  • A provider deprecating a model is disruptive even with a gateway abstraction, because the evaluation and any prompt tuning must be redone. It is planned for as a scheduled event twice a year.
◀  Governance and Audit All views
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 41 views · Microsoft Azure and open source41-failure-modes