SLO and Error Budget Service  ·  View 21 of 21  ·  Assurance

Failure Classes and Their Answers

Nine classes, each with the residual nobody designs away.

Editable source SVG draw.io All views
How it shows Detection Structural answer Accepted residual Telemetry gap Buckets stop arriving Coverage per SLO no-data + measurement alert Blind minutes unrecoverable Late data Bucket arrives at +12 min Lateness histogram Horizon, then coverage deficit Figure slightly pessimistic Poison input Implausible counter jump Tolerance on closed windows Quarantine, typed, replayable Coverage drops meanwhile Definition error Budget wrong, not missing Dry-run, shadow divergence Versioned rollback + recompute Verdicts served meanwhile Projection staleness Budget lags the buckets Watermark vs now Ceiling → insufficient-data Gate loses its input Verdict API down Gate cannot ask Independent watchdog Gate fails static on cache Policy weakens with age Alert storm 200 SLOs breach at once Pages per rotation Group by dependency, cap pages One real page may be grouped Regional loss Write region gone Health probes, watchdog Promote, then recompute 90 min of insufficient-data Privilege misuse Breach made to look met Exclusion minutes, override rate Two approvers, append-only Collusion defeats it Failure Classes and Their Structural Answers Nine classes, each with a residual nobody designs away. The two that would change the architecture are in the right-hand column: if blind minutes became common the platform would have to own collection after all, and if collusion on exclusions were plausible the figure would need an external witness. v 1.0 · owner Reliability Architecture · date 2026-10

The two that would change the design

  • If blind minutes became common rather than exceptional, the derivation boundary would stop paying for itself and the platform would have to own collection after all — which is the opposite of the critical design decision (ADR-02).
  • If collusion on exclusions were plausible in this organisation, no internal control would be sufficient and the published figure would need an external witness — a notarised snapshot or a third-party attestation.
  • Everything else in the table is handled structurally, with a residual that is stated rather than hidden.

Decisions

  • Every class resolves to publishing less rather than publishing something unsubstantiated. A reliability authority that produces a wrong number is worse than one that produces no number (ADR-04).
  • Quarantine, coverage deficit and insufficient-data are three different honest answers for three different failures, rather than one generic degraded state.
  • Regional loss is the only class whose answer is time rather than structure: 90 minutes of insufficient-data is the accepted cost of having one write region (ADR-16).

Assumptions

  • A tolerance on closed-window movement, beyond which an input batch is quarantined pending review — threshold assumed.
  • At most one re-run of a quarantined batch before a human looks at it.
  • The watchdog's own availability is not itself measured by this platform.
◀  Identity and Privilege Flow All views
Solution Architecture v1.0 · Microsoft Azure · Reliability Architecture · 2026-10 · 21 views · 17 architecture decision records21-failure-classes