Incident Management Platform  ·  View 21 of 34  ·  5 · Runtime

Alert Storm — 20,000 Alerts a Second for 120 Seconds

A collapsed dependency floods the platform with 2.4 million alerts. What each part does over the two minutes, and why the responder is woken once.

Editable source SVG draw.io All views
T0 · burst T + 5 s T + 60 s T + 120 s After Edge Noisy source over quota 429 to that source only Others admitted Source retries drain Shed count reported Ingest domain Burst buffered, R3 Consumer lag rises Lag bounded, alarmed Drained Incident engine One grouping key One incident opened Flood control attaches 2.4 M counted, 0 dropped Noise report entry Paging slice Reserved CPU untouched One page dispatched Open incidents on time No re-page Responder Woken once Sees live alert count Tunes the rule Alert Storm — 20,000 Alerts a Second for 120 Seconds Ingest and paging share hosts but not CPU: systemd slices cap the gateway and reserve the dispatcher, so a storm raises lag, never time to first notification. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Load is shed per integration at the edge, in HAProxy stick tables keyed by integration credential. The noisy integration gets 429s with a retry-after; every other integration is admitted at its normal rate.
  • Everything admitted is buffered, grouped and counted. Flood control attaches alerts to the open incident as counts rather than one event each, so the timeline shows 2.4 million alerts in one line and ClickHouse holds every one of them.
  • Paging capacity is reserved in CPU, not hoped for. The dispatcher's systemd slice has guaranteed cores that the gateway's slice cannot borrow.

Targets

  • 10× burst for 120 s with no increase in time to first notification for incidents already open. No alert dropped after admission. Ingest consumer lag bounded and alarmed at 60 s.

Proof

  • A recorded storm is replayed against the staging cells on every paging-path release (view 27). The pass condition is not throughput; it is the dispatch latency of an unrelated SEV2 injected in the middle of the storm.
◀  Provider Failover Within a Step All views Dependency Outages — What Still Works When Something Else Is Down  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises21-alert-storm