Observability Platform  ·  View 25 of 25  ·  Assurance

Failure Modes

Ten classes, each with how it is detected, contained, degraded and recovered — and one that looks like health.

Editable source SVG draw.io All views
Detected by Contained by Degrades to Recovers when Telemetry storm Ingest rate vs 5× budget Gateway admission + buffer depth Shed by declared class Backlog replays from offset Cardinality explosion Series admitted vs budget Per-service budget at the gateway Label dropped, aggregate kept Team fixes or takes an expiring exception Query stampede Frontend queue depth Per-tenant query admission Partial results, coverage stated Cache warms, incident ends Buffer unavailable Produce error rate Bounded gateway spool Shed by class, gap marked Brokers return; no catch-up flood Storage-writer lag Consumer lag, freshness p95 Buffer retention 72 h Freshness incident, lag shown in UI Writers scale, lag drains Query-tier outage Frontend 5xx, canary read fails Ruler on separate node groups No dashboards; paging unaffected Frontend restored; no data lost Control-plane outage Aurora health, Argo sync state Last-known config on every tier Fail static — no rule or budget changes Restored from Git, then Aurora Region loss Regional ingest to zero Regions collect and store locally Reduced replica only for that region Region returns; local data intact Platform is blind Dead-man switch, separate account Self-telemetry outside the domain Page the platform team directly Heartbeat resumes Silent gap Absence detection per producer Expected emission from the catalogue Gap marker written, never interpolated Producer resumes; gap stays on record Failure Modes — What Is Assumed To Fail, and What Happens The last row is the worst class because it looks like health. Every other row is detected by something going up; this one is detected by something that should be there and is not. v 1.0 · owner Reliability Architecture · date 2026-09

How to read it

  • Every row degrades to something named and visible. There is no row whose degradation is "data is missing and nobody knows" — that outcome is the one the architecture is built to eliminate.
  • Three rows fail static rather than failing closed: control-plane outage, buffer unavailability and query-tier outage all keep the platform serving on last-known state.

The last row

  • A silent gap is the worst class because every other failure is detected by something going up, and this one is detected by something that should be there and is not.
  • Absence detection per producer, driven by expected emission from the catalogue, is the only control that catches it — and it is the control most likely to be tuned down for noise.

Not covered here

  • Clock skew, poison payloads and credential compromise are named in the requirement (§11 of ask.md) and handled at the gateway and in view 23. They are omitted from this grid to keep it to ten rows that fit on one page.
◀  Identity and Access All views
Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views25-failure-modes