Incident Management Platform  ·  View 22 of 34  ·  5 · Runtime

Dependency Outages — What Still Works When Something Else Is Down

Six dependencies lost one at a time, against the four things responders need. Read down the first two columns: nothing in them is red.

Editable source SVG draw.io All views
Paging Acknowledgement Console Record Event log down Unaffected Unaffected Read-only, stale Outbox holds 30 d Control-plane cluster down Last snapshot · 72 h Unaffected Cell status page only Outbox holds 30 d Corporate IdP down Unaffected Unaffected · no login Break-glass keys Unaffected OpenBao down Keys held in memory Unaffected No secret rotation Unaffected Mattermost down Unaffected Unaffected Channel creation queued Unaffected Site A lost Cells B and C · quorum Anycast moves ≤ 2 min Rebuilt at B ≤ 15 min Sync standby · RPO 0 Dependency Outages — What Still Works When Something Else Is Down The degradation order is the requirement's: console first, then reviews, then analytics. No row reaches the paging column in red. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • When the control plane is down, each cell serves a read-only status page from its own state: open incidents, who was paged, who acknowledged. It is not the console. It is enough to run a response from.
  • Keys the paging path needs, carrier credentials and the contact-data key, are fetched from OpenBao at start and held in memory. An OpenBao outage stops rotation, not paging.
  • A degraded mode is announced. When a cell loses its connection to the control plane or the paging domain, the next page it sends says so in the incident summary, and the platform rotation is paged about it.

Degradation order

  • Exactly the requirement's: console first, then the review module, then analytics staleness. The paging path is not in the order, because nothing it needs sits below it.

Risks

  • A host restart during an OpenBao outage starts a dispatcher without keys. Hosts in the same cell restart one at a time and a starting dispatcher can request keys from a running peer over mTLS, so a single restart never leaves a cell blind.
◀  Alert Storm — 20,000 Alerts a Second for 120 Seconds All views Partition — A Cell Without Quorum Still Pages  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises22-dependency-outages