Incident Management Platform  ·  View 34 of 34  ·  7 · Assurance

Failure Modes — Detection, Containment, What Responders See

The requirement's ten named failure classes, each with how it is detected, what contains it, what a responder experiences and how the platform returns to normal.

Editable source SVG draw.io All views
Detected by Contained by Responders see Back to normal Carrier degraded Loopback SMS · call Route B within step Page on time Route health recovers Alert storm Shed and lag metrics Quota · flood control One page Noise report Coverage gap 14-day gap detector Fall to final step Manager paged at once Override added Unreachable human No ack in step Channels, then steps Secondary paged Ack from anyone Control plane down Synthetic · reconcile lag Snapshot · outbox Degraded banner Rebuild ≤ 15 min Estate sites lost Outpost probes Minority cell pages Possible duplicate Replay · labelled Poison alert Schema check Quarantine Nothing Mapping fixed Replayed webhook Source key · fingerprint Dedup · grouping CAS Nothing Counted DST or zone error Second resolver Old buckets held Nothing Owner fixes rule Escalation exhausted Terminal state Broadcast every 5 min Whole team paged Platform review Failure Modes — Detection, Containment, What Responders See v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Every row has a detection signal that exists before the failure does. A coverage gap is found 14 days out, a degraded carrier by loopback traffic, a DST error by the second resolver, not by a page that did not arrive.
  • The 'responders see' column is the design's promise. Where it is not 'nothing' or 'page on time', the cost is named: a manager paged early for a gap, a possible duplicate during a partition, a whole team paged when escalation is exhausted.

Drills

  • Quarterly: kill a cell, rebuild the control plane at site B, partition site C, exhaust a test policy, fail carrier A. Yearly: page a volunteer rotation with the manual fallback kit, a laptop, the exported snapshot and the GSM modem bank (ADR-30).

Residual risks

  • A simultaneous loss of both carriers in a responder's region, and a responder whose phone is off with no second device. Neither is solvable by the platform; both are why every policy ends in a broadcast to a team rather than a single person.
◀  Identity — Signing In When the Identity Provider Is the Outage All views
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises34-failure-modes