Incident Management Platform  ·  View 19 of 34  ·  5 · Runtime

Alert to Acknowledgement — The Core Loop

Thirteen messages from a signed webhook to a cancelled escalation, with every participant inside one paging cell except the source and the phone.

Editable source SVG draw.io All views
Monitoring source Ingest gateway Incident engine Coverage snapshot Escalation timers Dispatcher Responder phone Ack receiver 1. POST alert · HMAC 2. 202 · alert id 3. normalised + raw 4. dedup · CAS grouping key · rules 5. get rotation, current bucket 6. layers · verified methods 7. arm step 1 · write timer 8. fire step 1 9. push · critical alert 10. SMS at 30 s · voice at 2 min 11. DTMF 1 12. CAS state acked · cancel 13. withdraw pending sends Alert to Acknowledgement — The Core Loop Delivery receipts are written to the ledger and change nothing. Only the keypress, the SMS reply or the app button stops escalation. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • The gateway answers 202 with an alert id as soon as the message is durably in the ingest domain. The source's retry logic is then only exercised by genuine failures, and a retry that does arrive is absorbed by deduplication.
  • The timer is written before the first send. If the dispatcher crashes between arming and dispatching, the timer fires on another owner and the page still goes out, possibly twice, which the human-side dedup and Principle 3 both accept.
  • Acknowledgement is a compare-and-set on incident state in the paging domain. The timers and the dispatcher check that state before every fire and every send, so an acknowledgement stops queued work as well as future work.

Targets

  • Receipt to first dispatch: p95 ≤ 15 s, p99 ≤ 30 s. Keypress to escalation cancelled: p99 ≤ 5 s. Timer accuracy: ± 5 s at p99, including across a process restart.

Channel policy shown

  • SEV1: push immediately, SMS at 30 s, voice at 2 min, all inside a 5-minute step. SEV2: push and SMS immediately, voice at 5 min, 15-minute step. SEV3 and SEV4: push and email, no voice, which is where most of the spend is saved (view 29).
◀  The Log Is the Record — Events In, Projections Out All views Provider Failover Within a Step  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises19-alert-to-acknowledgement