Incident Management Platform  ·  View 20 of 34  ·  5 · Runtime

Provider Failover Within a Step

What the dispatcher does when a carrier refuses a message, and when a carrier accepts it and then goes quiet. The two cases get different answers.

Editable source SVG draw.io All views
Dispatcher SMS route A SMS route B Asterisk voice Responder Notification ledger 1. submit SMS 2. NACK · throttled 3. attempt 1 · failed 4. same SMS · route B 5. accepted · msg id 6. attempt 2 · accepted 7. no receipt in 20 s 8. call now, not at 2 min 9. rings · reads summary 10. presses 1 11. acknowledged 12. ack · step closed Provider Failover Within a Step — Same Channel on Failure, Next Channel on Silence A hard failure retries the same channel on the other route. Silence moves to a different channel, so a slow route never produces a second SMS inside 60 s. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • A hard failure, a NACK, a throttle or a refused connection, retries the same channel on the other route at once. The first message verifiably did not leave, so the retry cannot produce a duplicate.
  • Silence, an accepted message with no delivery receipt inside the channel's latency threshold, moves to the next channel rather than the other route. A slow route might still deliver, and a second SMS from route B would be the duplicate the 60-second rule forbids. A call is not a duplicate of an SMS (ADR-21).
  • Neither case advances the escalation step. Only the step timer does, because only a missing acknowledgement proves no human saw the page (ADR-16).

Thresholds

  • SMS receipt: 20 s. Voice: ringing within 10 s of originate. Push: accepted by APNs or FCM within 3 s. Route health is measured continuously by loopback traffic, so a degraded route is demoted before a real page tries it.

Risks

  • Delivery receipts are unreliable in both directions: some carriers report delivered on hand-off to the next network, some never report at all. The design never escalates on a receipt and never stops on one.
◀  Alert to Acknowledgement — The Core Loop All views Alert Storm — 20,000 Alerts a Second for 120 Seconds  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises20-provider-failover-within-a-step