Incident Management Platform

Architecture Views

34 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

An incident management platform for one organisation: 400 responders, 40 services, 25 on-call rotations across three timezones. It receives alerts, decides whether a human must be woken, works out which human, reaches them by push, SMS, voice and email, and keeps escalating until someone acknowledges. Around that loop it keeps an immutable incident timeline, coordinates the response and produces the post-incident review. It is built from open-source software on hardware the organisation owns: HAProxy and a Go ingest gateway at the edge, NATS JetStream for paging state stretched across three sites, Asterisk and Jasmin on two independent carrier routes, PostgreSQL for the append-only event log, ClickHouse for the notification ledger and noise reporting, Keycloak and OpenBao for identity and secrets. Read the set in order. Acts 1 and 2 fix the boundary and the people the platform wakes. Act 3 separates the paging path from everything else and follows one page through it. Act 4 separates four stores by what may be lost. Act 5 walks the core loop, a provider failure, an alert storm, a partition, a DST night and escalation exhaustion. Acts 6 and 7 cover where it runs, how it changes, how it is watched and paid for, then what could go wrong. One boundary governs every page: the paging path can decide, dispatch and accept an acknowledgement with the incident store, the console, the identity provider and the control plane all unavailable.

1 · Context and scope

What the platform owns between a signal and an acknowledged human, what it reaches, and what it borrows but must survive losing.
01 Signals it receives Monitoring and alerting metric · log · synthetic Email and ticket sources vendors · helpdesk People it serves and informs On-call responder Incident commander Rotation owner Service owner Stakeholders What it reaches Voice and SMS carriers two, independent Apple and Google push APNs · FCM Mattermost · tracker channels · action items Incident Management Platform 400 responders · 25 rotations What it borrows, and must survive losing Corporate identity provider federated only Delivery pipelines deploy · change events signed webhooks SMTP · webhook acknowledges coordinates schedules · overrides reviews · noise updates · status page SIP · SMPP critical alerts channels · actions SSO change events Incident Management Platform — System Context External / third party Person or role synchronous event / async v 1.0 · owner Reliability Architecture · date 2026-09 System Context The signals the platform consumes, the people it serves and informs, what it reaches to wake a human, and the two dependencies it borrows but must survive losing. HTML page SVG draw.io
02 Receive HAProxy edge anycast · quotas Ingest gateway verify · normalise Decide Incident engine dedup · group · suppress Coverage snapshot JetStream KV Page Escalation timers durable · ± 5 s Dispatcher channel policy Reach Asterisk · Jasmin two carrier routes Push relay APNs · FCM Confirm Ack receiver every channel Notification ledger per attempt Record · control plane Incident event log PostgreSQL · append-only Schedule resolver 72 h ahead snapshot ≤ 60 s High-Level Architecture — Two Planes, One Loop Interface / broker Application we own Data store event / async The first five stages are the paging path and run with the sixth stage entirely unavailable. The only arrow into it from the control plane is a replicated snapshot. v 1.0 · owner Reliability Architecture · date 2026-09 High-Level Architecture — Two Planes, One Loop The core loop from receipt to acknowledgement as five stages that form the paging path, and the sixth stage, the record and the control plane, which the loop does not wait for. HTML page SVG draw.io

2 · People and journeys

Who the platform wakes, who it answers to, and the four moments where it either earns trust or spends it.
03 People woken by it On-call responder ≈ 400 across 25 rotations Goal — Wake me only for something real, and let one keypress stop the pager. Core journeys Acknowledge a SEV1 at 03:12 Hand over at the end of a shift Pull a colleague in by role Incident commander every SEV1 and SEV2 Goal — Get the right people in one place, and keep stakeholders informed without letting them into the channel. Core journeys Run a SEV1 to mitigation Split a second failure out Post a stakeholder update People who own what it pages for Rotation owner 25 rotations · 3 timezones Goal — Hear about the hole in my rotation a week before it matters, not when a page fails. Core journeys Close a coverage gap before DST Approve a holiday override Chase an unverified phone number Service owner 40 services Goal — Pay less for noise, and make the review the least effort of anything I do that week. Core journeys Close a review and its actions Read the weekly noise and cost report Declare a maintenance window Machines and readers Monitoring system Alertmanager · synthetics Goal — Hand an alert over once, retry safely, and be throttled rather than dropped. Core journeys Retry a webhook after a timeout Survive its own quota Stakeholder support · leadership Goal — Know what customers see and when I will hear next, without joining the incident. Core journeys Read a templated update Check the status page Actors — Who the Platform Is For and What They Get to Do Person or role Journey / task Security / platform External / third party v 1.0 · owner Reliability Architecture · date 2026-09 Actors — Who the Platform Is For and What They Get to Do Six actors grouped by their relationship to the platform, each with the goal in their own words and the journeys that goal produces. Four of those journeys are mapped on the next pages. HTML page SVG draw.io
04 On-call responder payments primary Goal — Be reached, understand why, act Trigger — checkout error rate at 18%, SEV1 proposed Done when — Acknowledged in 3 min, rollback started 1 · Alert fires 2 · Push 3 · Call rings ◆ moment of truth 4 · Acknowledge 5 · Orient 6 · Mitigate What they do Asleep, phone on DND Sleeps through it Answers the call Presses 1 Opens the incident link Rolls back the deploy System behind it Engine · snapshot Push relay Asterisk · carrier A Ack receiver Timeline · catalogue Change events How it feels Calm Groggy Stressed Where it hurts Push swallowed? Why me, what broke? Where is the runbook? What the platform gives Right human, one lookup Critical alert for SEV1 Service named in speech Escalation off ≤ 5 s Runbook from catalogue Deploy on the timeline Journey — An On-Call Responder Is Woken for a SEV1 at 03:12 v 1.0 · owner Reliability Architecture · date 2026-09 Journey — An On-Call Responder Is Woken for a SEV1 at 03:12 Six phases from alert to rollback. The trough is the call: the only moment the platform must beat a phone on do-not-disturb and a responder who slept through the push. HTML page SVG draw.io
05 Incident commander checkout outage Goal — Stop user impact, keep everyone informed Trigger — Responder raises the incident to SEV1 Done when — Mitigated in 41 min, updates on time 1 · Declare 2 · Assemble 3 · Organise 4 · Split ◆ moment of truth 5 · Update 6 · Mitigated What they do Takes the IC role Pages database team Names scribe and comms Splits the DNS alerts out Posts the 30-min update Marks mitigated System behind it Severity obligations Service catalogue Role events Split event Update template Three clocks How it feels Calm Tense Stressed Where it hurts Who owns payments-db? Twenty people typing Second failure hidden Leaders asking in DMs What the platform gives Channel opened for SEV1 Page by service owner One holder per role New service re-pages Next-update-by timer Mitigated is not resolved Journey — An Incident Commander Runs a SEV1 to Mitigation v 1.0 · owner Reliability Architecture · date 2026-09 Journey — An Incident Commander Runs a SEV1 to Mitigation From declaration to mitigated in 41 minutes. The trough is the split: a second, unrelated failure hiding inside an incident everyone has already acknowledged. HTML page SVG draw.io
06 Rotation owner platform-db · US/Eastern Goal — No hour without a responder Trigger — Gap warning for the spring-forward night Done when — Covered, verified, provable later 1 · Warned 2 · Inspect 3 · Override 4 · Verify 5 · The night ◆ moment of truth 6 · Prove What they do Reads the gap warning Opens coverage preview Adds a 01:00–04:00 override Reads the snapshot diff Sleeps Answers who was on call System behind it Gap detector · 14 d Resolved timeline Override layer Second resolver Snapshot · 72 h ahead Schedule audit How it feels Confident Unsure Anxious Where it hurts 02:30 does not exist Four layers deep Will anyone be paged? What the platform gives Warned 9 days early Shown in local time Override beats all layers Two resolvers agree Nothing computed live Answer from the record Journey — A Rotation Owner Closes a Gap Nine Days Before DST v 1.0 · owner Reliability Architecture · date 2026-09 Journey — A Rotation Owner Closes a Gap Nine Days Before DST A US/Eastern rotation hands over at 02:30 local, a time that does not exist on the spring-forward night. The platform finds it nine days out, not when a page falls through. HTML page SVG draw.io
07 Service owner search-api Goal — A review that changes something Trigger — SEV2 resolved, review due in 5 business days Done when — Reviewed on day 3, actions closed with change 1 · Draft 2 · Timeline 3 · Factors 4 · Actions ◆ moment of truth 5 · Follow up 6 · Recur What they do Opens the draft Checks the intervals Writes conditions Files three actions Chases the overdue one Searches past reviews System behind it Review generator Event log Blameless template Tracker sync Overdue report Factor search How it feels Confident Unsure Reluctant Where it hurts Another form to fill Urge to name a person Items die in trackers Closed, nothing changed What the platform gives Pre-filled on resolve Intervals computed No field for a name Two-way sync No-change closures flagged Same factor, third time Journey — A Service Owner Turns a SEV2 Into Fixed Conditions v 1.0 · owner Reliability Architecture · date 2026-09 Journey — A Service Owner Turns a SEV2 Into Fixed Conditions The review is where the platform either reduces next quarter's pages or produces paperwork. The trough is the action items, which is where most review programmes quietly die. HTML page SVG draw.io

3 · Structure

Two planes with different availability budgets, one paging cell, the control plane, every interface, one page end to end, and the order of the checks before the platform decides not to page.
08 Signal edge HAProxy edge BGP anycast · quotas Postfix inbound email ingest · replies Ingest gateway Go · CEL mappings Quarantine per integration Paging path Incident engine dedup · group · suppress Escalation timers sharded wheel Dispatcher channel policy Ack receiver DTMF · SMS · push Paging state NATS JetStream stretched · 3 sites Coverage snapshot KV · 72 h ahead Notification ledger stream · 30 d Delivery Push relay APNs · FCM Jasmin SMS SMPP · two routes Asterisk SIP · two carriers Postfix outbound two smarthosts Control plane Schedule resolver IANA zones Incident service roles · updates Review module drafts · actions Console web · break-glass Record and insight PostgreSQL event log · projections ClickHouse ledger · raw · noise Ceph RGW Object Lock archive Identity, secrets and reporting Keycloak OIDC broker OpenBao secrets · transit keys Grafana reports · dashboards normalised call · SMS append Layered Architecture Interface / broker Application we own Risk / gap Queue / topic Data store Security / platform event / async synchronous The top four layers are the paging path and depend on nothing below them at run time. The control plane reaches up only by publishing a snapshot into the paging state. v 1.0 · owner Reliability Architecture · date 2026-09 Layered Architecture Seven layers. The top four are the paging path; the bottom three can be unavailable without a page being lost. HTML page SVG draw.io
09 Paging cell · site A · three dedicated hosts · systemd, no orchestrator Receive HAProxy edge anycast VIP · BFD Ingest gateway ingest.slice · capped State Ingest domain JetStream · site R3 Paging domain JetStream · 3-site R3 Decide and page · paging.slice, reserved CPU Incident engine grouping key CAS Escalation timers shard lease Dispatcher human dedup 60 s Ack receiver signed ack tokens Delivery Asterisk IVR · DTMF Jasmin SMS SMPP · MO replies Push relay APNs · FCM Monitoring sources signed webhooks Cells B and C Raft peers Carrier A SIP trunk · SMPP HTTPS publish CAS · outbox SIP · TLS Raft Container Architecture — One Paging Cell Interface / broker Application we own Queue / topic External / third party synchronous event / async Omitted for legibility: the engine consuming the ingest domain, timers and dispatcher reading the paging domain, and every ack path (view 19). v 1.0 · owner Reliability Architecture · date 2026-09 Container Architecture — One Paging Cell Everything that runs in one cell: three dedicated hosts at one site, with no orchestrator, no shared DNS and no registry needed at run time. Cells B and C have the same shape. HTML page SVG draw.io
10 Control plane · site A · platform-owned Kubernetes, warm rebuild at site B Response Incident service state · roles · updates Console TypeScript web UI Review module drafts · actions Scheduling Service catalogue YAML in Git Schedule resolver 72 h · 14 d gaps DST verifier second implementation Data PostgreSQL CloudNativePG · sync Projector rebuildable views ClickHouse ledger · noise · cost Integration and identity Keycloak broker · break-glass Chat bridge channel per incident Tracker sync two-way · reconciled Paging cells coverage KV · outbox Corporate IdP OIDC · SAML publish snapshot append federate Container Architecture — The Control Plane Application we own Decision point Data store Security / platform Interface / broker External / third party event / async synchronous Omitted for legibility: the reconciler reading the paging outbox (view 18), the chat bridge and tracker sync reaching Mattermost and the tracker (view 11), and OpenBao, which every service reads at start. v 1.0 · owner Reliability Architecture · date 2026-09 Container Architecture — The Control Plane The services that edit schedules, run the response, hold the record and write reviews. It runs on a small Kubernetes cluster the platform team owns, at one site, with a warm rebuild at the other. HTML page SVG draw.io
11 What sends to it Monitoring sources Email · change feed Automation clients Platform Incident Management Platform decide · page · record What it reaches Carriers · APNs · FCM Mattermost · tracker Outbound event stream HMAC webhooks SMTP · events REST · on-call lookup SIP · SMPP · push channels · actions lifecycle events Integration Surface — Every Interface External / third party Application we own Queue / topic synchronous event / async Only the left column and the carriers sit on the paging path. Everything else on the right is allowed to be late. v 1.0 · owner Reliability Architecture · date 2026-09 Integration Surface — Every Interface What sends to the platform, what the platform reaches, and which of those interfaces sit on the paging path. HTML page SVG draw.io
12 Admit Integration quota HAProxy stick table Signature check HMAC · two secrets Normalise CEL mapping per integration Fingerprint source · key · fp Decide Dedup window 24 h · site KV Grouping key CAS one open incident Resolve Coverage bucket rotation · 15 min Escalation policy versioned KV Dispatch Human dedup 60 s per channel Channel policy per severity Confirm Acknowledgement cancels step ≤ 5 s Ledger entry per attempt Anatomy of a Page — Every Step Local to One Cell Decision point Security / platform Application we own Data store Budget for alert receipt to first dispatch: p95 15 s. Nothing on this page makes a network call outside the cell except the carrier submit. v 1.0 · owner Reliability Architecture · date 2026-09 Anatomy of a Page — Every Step Local to One Cell The twelve decisions between an alert arriving and a ledger entry being written, in order, with the only external call being the carrier submit. HTML page SVG draw.io
13 Seen before? Exact duplicate source key · 24 h Recorded, not paged duplicate count Exempt? SEV1 or declared by a human Bypasses suppression still grouped Maintenance? Declared window owner · expiry Recorded, visible rule id logged Already open? Open incident same grouping key Attach, no re-page unless severity rises Flooding? Rate per key over threshold Attach, count never dropped Page Escalation starts policy step 1 Decision event rule that decided May the Platform Decide Not to Page? — The Order of the Checks Decision point Data store Opportunity Application we own Every box in the lower row writes an event naming the rule that decided. Dependency-aware suppression is absent on purpose; it annotates, it never withholds. v 1.0 · owner Reliability Architecture · date 2026-09 May the Platform Decide Not to Page? The fixed order of the five checks that can stop a page, what happens instead at each, and why the most tempting check of all is not on the page. HTML page SVG draw.io

4 · Data

Four stores separated by what may be lost and what must be fast, the data model, and the three pipelines that move alerts in, coverage out and events into the record.
14 Platform state, by recovery obligation Coverage · RPO 0 Coverage snapshot JetStream KV · R3 Escalation policies KV · versioned Schedule store PostgreSQL · source Event log · system of record · 7 y Paging outbox JetStream · 30 d Incident event log PostgreSQL · append-only Chain anchors Ceph · Object Lock Notification ledger · 13 mo Ledger stream JetStream · 30 d Ledger archive ClickHouse · 13 mo Operational · rebuildable Projections PostgreSQL schema Review search full-text index Evidence · loss-tolerant Raw payloads ClickHouse · 90 d Noise and cost marts ClickHouse · 13 mo Four Stores, Separated by What May Be Lost and What Must Be Fast The first column is small and must be current. The second must never change. The third must survive the second. The last two may be rebuilt or lost. v 1.0 · owner Reliability Architecture · date 2026-09 Four Stores, Separated by What May Be Lost and What Must Be Fast The requirement's four stores plus the evidence that feeds reporting, arranged by recovery obligation rather than by technology. HTML page SVG draw.io
15 integration integration_id PK owner_team quota_per_s secret_ref two active mapping_version alert alert_id PK integration_id FK source_key, fingerprint proposed_severity source_ts, received_ts raw_ref 90 d review review_id PK incident_id FK contributing_factors system_conditions due_at 5 business days state action_item action_id PK review_id FK owner_team, due_at tracker_ref state closed_with_change service service_id PK owner_team policy_id FK runbook_url depends_on incident incident_id PK service_id FK grouping_key state, severity acked_at, mitigated_at resolved_at restricted bool incident_event event_id PK idempotent incident_id FK seq, received_ts type, actor payload prev_hash, hash escalation_policy policy_id, version PK steps targets, wait_s final_step NOT NULL channel_policy notification_attempt attempt_id PK incident_id, step channel, provider provider_msg_id dispatched_at, outcome acked_at, cost contact_method method_id PK responder_id type push|sms|voice|email address encrypted verified_at rotation rotation_id PK iana_zone owner_team handoff_local_time coverage_bucket rotation_id, bucket_utc PK layers ordered method_ids verified only snapshot_version verified_by both resolvers schedule_layer layer_id PK rotation_id FK kind base|secondary|mgmt|override rule or window changed_by, changed_at 1 : N N : 1 1 : N 1 : N cites 1 : N 1 : N 1 : N N : 1 N : M 1 : N N : M 1 : N Incidents, Coverage and Evidence — Data Model incident is a projection of incident_event. contact_method.address is encrypted under a per-responder key; the event log holds only responder ids. v 1.0 · owner Reliability Architecture · date 2026-09 Incidents, Coverage and Evidence — Data Model Thirteen entities across alerting, response, scheduling and evidence, with the keys that make ingest idempotent, coverage one lookup and the timeline tamper-evident. HTML page SVG draw.io
16 Sources Alertmanager grouped upstream Vendor email SMTP Edge HAProxy edge quota · 429 Postfix inbound per-integration address Gateway Verify HMAC · replay window Normalise CEL · schema check Buffer Raw and normalised one message, both Quarantine schema failures Consume Incident engine per-integration consumer Quarantine monitor threshold · SEV4 Keep Raw payloads ClickHouse · 90 d Integration owner low-severity incident HTTPS SMTP parsed valid invalid sink over threshold Alert Ingest — Untrusted In, Normalised Out, Nothing Lost External / third party Interface / broker Security / platform Application we own Queue / topic Risk / gap Decision point Data store Person or role synchronous event / async failure / alternate batch Source and receive timestamps are both kept and never reconciled. A malformed payload is kept and counted, never rejected silently. v 1.0 · owner Reliability Architecture · date 2026-09 Alert Ingest — Untrusted In, Normalised Out, Nothing Lost Two transports into one normalised stream, with the raw payload travelling beside every normalised alert and every failure kept and counted. HTML page SVG draw.io
17 Edit Schedule editor rotation owners only Override wins every layer Store Schedule store PostgreSQL Schedule audit actor · window Resolve Schedule resolver IANA zones · 72 h Gap detector 14 d horizon Verify DST verifier independent code Diff gate hold on disagreement Publish Coverage buckets 15 min · R3 quorum Snapshot version stamped on pages Page Dispatcher one KV get Staleness alarm p99 ≤ 60 s Coverage Snapshot — Resolved Ahead, Verified Twice, Read in One Lookup Application we own Data store Decision point Security / platform A change is visible to paging within 60 s at p99. If the two resolvers disagree, the previous verified buckets keep serving and the owner is told why. v 1.0 · owner Reliability Architecture · date 2026-09 Coverage Snapshot — Resolved Ahead, Verified Twice, Read in One Lookup How layered rotations, overrides and timezones become a 15-minute bucket that the dispatcher reads with a single key, and what stops a wrong answer from reaching it. HTML page SVG draw.io
18 Emit Paging outbox cells A · B · C Console commands notes · severity · roles Reconcile Reconciler idempotent on event_id Minority replay after partition Append Incident event log no UPDATE · no DELETE Hash chain per incident Project Projector replayable Operational store open incidents · views Serve Console p95 ≤ 1.5 s Outbound stream idempotency keys Preserve Daily anchors Ceph · Object Lock Rebuild drill quarterly · ≤ 60 min The Log Is the Record — Events In, Projections Out Queue / topic Application we own Decision point Data store Security / platform A correction is a new event that supersedes an old one. Merge and split are events too, so both sides of either keep their original timelines. v 1.0 · owner Reliability Architecture · date 2026-09 The Log Is the Record — Events In, Projections Out How events from three paging cells and the console become one ordered, append-only timeline, and how every view of an incident is derived from it. HTML page SVG draw.io

5 · Runtime

The core loop, a provider failing mid-step, a 20,000-per-second storm, dependency outages, a partitioned cell, a DST night and escalation exhaustion.
19 Monitoring source Ingest gateway Incident engine Coverage snapshot Escalation timers Dispatcher Responder phone Ack receiver 1. POST alert · HMAC 2. 202 · alert id 3. normalised + raw 4. dedup · CAS grouping key · rules 5. get rotation, current bucket 6. layers · verified methods 7. arm step 1 · write timer 8. fire step 1 9. push · critical alert 10. SMS at 30 s · voice at 2 min 11. DTMF 1 12. CAS state acked · cancel 13. withdraw pending sends Alert to Acknowledgement — The Core Loop Delivery receipts are written to the ledger and change nothing. Only the keypress, the SMS reply or the app button stops escalation. v 1.0 · owner Reliability Architecture · date 2026-09 Alert to Acknowledgement — The Core Loop Thirteen messages from a signed webhook to a cancelled escalation, with every participant inside one paging cell except the source and the phone. HTML page SVG draw.io
20 Dispatcher SMS route A SMS route B Asterisk voice Responder Notification ledger 1. submit SMS 2. NACK · throttled 3. attempt 1 · failed 4. same SMS · route B 5. accepted · msg id 6. attempt 2 · accepted 7. no receipt in 20 s 8. call now, not at 2 min 9. rings · reads summary 10. presses 1 11. acknowledged 12. ack · step closed Provider Failover Within a Step — Same Channel on Failure, Next Channel on Silence A hard failure retries the same channel on the other route. Silence moves to a different channel, so a slow route never produces a second SMS inside 60 s. v 1.0 · owner Reliability Architecture · date 2026-09 Provider Failover Within a Step What the dispatcher does when a carrier refuses a message, and when a carrier accepts it and then goes quiet. The two cases get different answers. HTML page SVG draw.io
21 T0 · burst T + 5 s T + 60 s T + 120 s After Edge Noisy source over quota 429 to that source only Others admitted Source retries drain Shed count reported Ingest domain Burst buffered, R3 Consumer lag rises Lag bounded, alarmed Drained Incident engine One grouping key One incident opened Flood control attaches 2.4 M counted, 0 dropped Noise report entry Paging slice Reserved CPU untouched One page dispatched Open incidents on time No re-page Responder Woken once Sees live alert count Tunes the rule Alert Storm — 20,000 Alerts a Second for 120 Seconds Ingest and paging share hosts but not CPU: systemd slices cap the gateway and reserve the dispatcher, so a storm raises lag, never time to first notification. v 1.0 · owner Reliability Architecture · date 2026-09 Alert Storm — 20,000 Alerts a Second for 120 Seconds A collapsed dependency floods the platform with 2.4 million alerts. What each part does over the two minutes, and why the responder is woken once. HTML page SVG draw.io
22 Paging Acknowledgement Console Record Event log down Unaffected Unaffected Read-only, stale Outbox holds 30 d Control-plane cluster down Last snapshot · 72 h Unaffected Cell status page only Outbox holds 30 d Corporate IdP down Unaffected Unaffected · no login Break-glass keys Unaffected OpenBao down Keys held in memory Unaffected No secret rotation Unaffected Mattermost down Unaffected Unaffected Channel creation queued Unaffected Site A lost Cells B and C · quorum Anycast moves ≤ 2 min Rebuilt at B ≤ 15 min Sync standby · RPO 0 Dependency Outages — What Still Works When Something Else Is Down The degradation order is the requirement's: console first, then reviews, then analytics. No row reaches the paging column in red. v 1.0 · owner Reliability Architecture · date 2026-09 Dependency Outages — What Still Works When Something Else Is Down Six dependencies lost one at a time, against the four things responders need. Read down the first two columns: nothing in them is red. HTML page SVG draw.io
23 Outpost synthetic probe Cell C engine Paging domain Cell C dispatcher Carrier B Responder Reconciler 1. estate checks failing 2. CAS open incident 3. no quorum · 1 of 3 4. minority mode · local log 5. page from last snapshot 6. SMS and voice · route B 7. rings 8. ack by SMS reply 9. quorum restored 10. replay minority log 11. mark duplicates · keep both Partition — A Cell Without Quorum Still Pages Quorum decides what the record says, never whether a human is paged. The cost is a possible duplicate page, which the reconciler labels rather than hides. v 1.0 · owner Reliability Architecture · date 2026-09 Partition — A Cell Without Quorum Still Pages Sites A and B disappear from site C's point of view. The outpost's probes see the estate failing, its paging domain has no quorum, and it pages anyway. HTML page SVG draw.io
24 T − 14 d T − 9 d T − 72 h 02:00 local After Resolver Handoff at 02:30 local Instant does not exist Buckets materialised Verifier Second resolver disagrees Both agree after fix Rotation owner Gap warning received Override 01:00–04:00 Asleep Snapshot Old buckets held Verified buckets live Served unchanged Paging Page lands on override Audit names who A DST Boundary — Found Nine Days Early, Not at 02:30 Nothing about the clock change is computed on the night. The resolver's rule for a missing local time is to hand over at the next valid instant, and the verifier checks it did. v 1.0 · owner Reliability Architecture · date 2026-09 A DST Boundary — Found Nine Days Early, Not at 02:30 A rotation's handoff time does not exist on the spring-forward night. Two resolvers disagree about what that means, and that disagreement is the warning. HTML page SVG draw.io
25 Escalation timers Dispatcher Primary Secondary Management layer Owning team Platform rotation 1. step 1 · 5 min 2. push · SMS · voice 3. no ack · step 2 4. push · SMS · voice 5. primary keeps ringing 6. no ack · final step 7. voice · every channel 8. broadcast to whole team 9. state: exhausted 10. platform SEV2 · route B 11. repeat broadcast every 5 min Escalation Exhausted — The Worst Outcome Is Loud Exhausted is a state, not an ending. It stays open, keeps broadcasting, and opens an incident against the platform, because either the policy or the rota is wrong. v 1.0 · owner Reliability Architecture · date 2026-09 Escalation Exhausted — The Worst Outcome Is Loud Nobody acknowledged through three steps. The policy's final step is a broadcast, and reaching it opens an incident against the platform. HTML page SVG draw.io

6 · Operations

Where it runs, how the paging path changes safely, how the watcher is watched, what it costs in money and in sleep, the build order and the loop it closes.
26 Three locations, three power feeds, two upstream carriers Site A · data centre Paging cell A · 3 hosts NATS JetStream ingest R3 · paging voter Paging services engine · timers · dispatch Asterisk · Jasmin carrier A Control plane · platform Kubernetes Control services incident · schedule · review PostgreSQL primary CloudNativePG Site B · data centre Paging cell B · 3 hosts NATS JetStream ingest R3 · paging voter Paging services active-active with A Asterisk · Jasmin carrier A Warm control plane Argo CD manifests scaled to zero PostgreSQL standby synchronous Site C · colocation outpost Paging cell C · 3 hosts NATS JetStream paging voter Paging services minority-capable Asterisk · Jasmin carrier B · GSM bank Outpost-only duties Synthetic and watchdog dead-man switch Status page nginx · static Deployment — Two Estate Sites and an Outpost That Shares Neither Queue / topic Application we own Interface / broker Data store Security / platform The estate runs in A and B. C runs nothing the estate needs and needs nothing the estate runs: own ISP, own DNS resolvers, own time source. v 1.0 · owner Reliability Architecture · date 2026-09 Deployment — Two Estate Sites and an Outpost That Shares Neither Three paging cells in three locations, the control plane at site A with a warm rebuild at site B, and the duties only the outpost performs. HTML page SVG draw.io
27 Source Forgejo signed commits · 2 reviewers Forgejo Actions static Go binaries Prove Schedule tests every DST to 2030 Storm replay recorded 20k/s burst Sign cosign · SBOM Harbor OCI artefact Staging cells synthetic pages Gate Change freeze check no open SEV1 or SEV2 Business hours both regions awake Cell C Ansible · cell C host by host Synthetic gate 3 pages acked Cells B then A Ansible · cell B next day Ansible · cell A day after fail Changing the Paging Path — One Cell at a Time, Never During a SEV1 Data store Application we own Decision point Security / platform failure / alternate Binaries and configuration are staged on every host before a switch, so a deploy needs no registry, DNS or Git reachable at the moment it happens. v 1.0 · owner Reliability Architecture · date 2026-09 Changing the Paging Path — One Cell at a Time, Never During a SEV1 The route a change to the paging path takes from commit to the third cell, and the gates that decide whether it moves on. HTML page SVG draw.io
28 Ingest Decide Dispatch Acknowledge Record Metrics Accepted · shed Suppression rate Receipt latency Time to ack Reconcile lag Freshness Consumer lag Snapshot age Timer drift p99 Cancel latency p99 Projection age Synthetic Test alert · 60 s Test incident opened SMS and call to own SIM Robot presses 1 Event appears in log Outside view Carrier B only Dead-man switch Observability — Watching the Watcher From Outside The platform's own alerts are paged through itself and, independently, by the outpost watchdog on carrier B. A silent platform is the alert. v 1.0 · owner Reliability Architecture · date 2026-09 Observability — Watching the Watcher From Outside What is measured at each stage of the loop, which signals are about freshness rather than volume, and how a silent platform is detected by something that is not the platform. HTML page SVG draw.io
29 Measured from Attributed to Reported Lever Voice minutes Ledger · carrier rate card Service · rotation Weekly · with noise Push first below SEV2 SMS Ledger · per segment Service · rotation Weekly · with noise Fix the noisiest rule Outpost standby Colocation · ISP · trunks Platform budget Yearly · stated up front Insurance, not waste Synthetic traffic Ledger · test target Platform budget Monthly Interval per channel Out-of-hours pages Ledger · responder zone Responder · rotation Weekly · target ≤ 2 Rotation size · rules Actionable rate Ack then change or note Service · alert rule Weekly · target ≥ 70% Demote or delete rule Cost and Human Load — What Gets Measured, Charged and Reported Human cost is measured and reported, never enforced by suppression. A rotation over budget gets a report, not fewer pages. v 1.0 · owner Reliability Architecture · date 2026-09 Cost and Human Load — What Gets Measured, Charged and Reported The platform's marginal cost is carrier traffic, driven by alert quality; its human cost is interrupts, driven by the same thing. Both are measured per service and reported together. HTML page SVG draw.io
30 Prove · 8 weeks Stretched JetStream 3-site latency Carrier routes receipts · DTMF Critical alerts iOS entitlement MVP Ingest · dedup · raw webhooks Schedules · snapshot gaps · DST verifier Escalation · 4 channels two providers · acks Event log · synthetic break-glass · APIs Phase 2 Grouping · merge · split maintenance · flood Channels · roles stakeholder updates Reviews · tracker sync computed intervals Outbound stream · cost status page Phase 3 Dependency hints annotate, never withhold Interrupt analytics rotation fairness Regional failover drills automated Sensitive incidents factor search · fallback kit Roadmap — What Ships First, and Why in That Order Decision point Risk / gap Application we own Security / platform The MVP already runs across three sites. Independence is not a Phase 3 feature, because a paging path that shares the estate's failure is not an MVP of this platform. v 1.0 · owner Reliability Architecture · date 2026-09 Roadmap — What Ships First, and Why in That Order An eight-week proof, then the requirement's MVP, Phase 2 and Phase 3, with one deliberate change: independence from the estate is in the MVP, not deferred. HTML page SVG draw.io
31 Signal alert or a person Page right human Acknowledge one keypress Mitigate impact stops Review conditions, not people Action items closed with change Alert hygiene noise · cost report Incident decided reached coordinated resolved tracked tuned fewer, truer The Loop the Platform Exists to Close External / third party Application we own Decision point Data store Security / platform v 1.0 · owner Reliability Architecture · date 2026-09 The Loop the Platform Exists to Close Seven stages from a signal to fewer, truer signals. The platform is judged on the right half of the ring as much as on the left. HTML page SVG draw.io

7 · Assurance

Trust zones, signing in when the identity provider is the outage, and every named failure class with what contains it.
32 Untrusted senders Integrations one credential each Responder devices app · SMS · voice Edge Ingest gateway verify before parse Ack receiver signed, single-use Paging cells Incident engine mTLS between cells Dispatcher decrypts at send Key custody OpenBao secrets · transit Responder keys deleted on offboarding Control plane Console team-scoped RBAC Event log row-level security HMAC · TLS ack token normalised unwrap data key short-lived creds Security — Trust Zones and What Crosses Them External / third party Security / platform Application we own Data store synchronous event / async An integration's credential can only submit alerts for itself. A leaked ack token stops one escalation once; it cannot read an incident. v 1.0 · owner Security Architecture · date 2026-09 Security — Trust Zones and What Crosses Them Five zones from untrusted senders to key custody, and the one credential or token that each crossing requires. HTML page SVG draw.io
33 Incident commander Console Keycloak Corporate IdP Hardware key Audit events Security rotation 1. open console 2. OIDC redirect 3. federated login 4. timeout 5. offer break-glass 6. local account 7. WebAuthn challenge 8. signed assertion 9. 4 h session · responder role 10. break-glass used 11. notify, never page Identity — Signing In When the Identity Provider Is the Outage Break-glass accounts can acknowledge, resolve and edit incidents. They cannot edit schedules, policies or suppression rules, which can wait for the IdP. v 1.0 · owner Security Architecture · date 2026-09 Identity — Signing In When the Identity Provider Is the Outage Federated login fails because the corporate identity provider is down. An incident commander still reaches the console, with less power and more scrutiny. HTML page SVG draw.io
34 Detected by Contained by Responders see Back to normal Carrier degraded Loopback SMS · call Route B within step Page on time Route health recovers Alert storm Shed and lag metrics Quota · flood control One page Noise report Coverage gap 14-day gap detector Fall to final step Manager paged at once Override added Unreachable human No ack in step Channels, then steps Secondary paged Ack from anyone Control plane down Synthetic · reconcile lag Snapshot · outbox Degraded banner Rebuild ≤ 15 min Estate sites lost Outpost probes Minority cell pages Possible duplicate Replay · labelled Poison alert Schema check Quarantine Nothing Mapping fixed Replayed webhook Source key · fingerprint Dedup · grouping CAS Nothing Counted DST or zone error Second resolver Old buckets held Nothing Owner fixes rule Escalation exhausted Terminal state Broadcast every 5 min Whole team paged Platform review Failure Modes — Detection, Containment, What Responders See v 1.0 · owner Reliability Architecture · date 2026-09 Failure Modes — Detection, Containment, What Responders See The requirement's ten named failure classes, each with how it is detected, what contains it, what a responder experiences and how the platform returns to normal. HTML page SVG draw.io

Architecture One-Pager

The problem, the shape of the answer, the decisions that carry it, and why it should still be right in ten years.

A paging path that can wake the right person with the incident store, the console, the identity provider and both estate data centres gone, and a record that tells the truth about what happened afterwards.

Four hundred engineers are on call for forty services. When something breaks, a signal has to become a phone ringing in the right pocket within seconds, and it has to keep ringing, in more pockets, until somebody says they have it. That is a small system by traffic. It is a hard system by circumstance: it is used precisely when other systems are failing, and very often when the systems it runs on are the ones failing. A platform built like an ordinary internal application, one database, the shared Kubernetes cluster, corporate single sign-on, will work every day except the days it exists for. The second problem is trust. A platform that pages for noise teaches people to ignore it, and a platform that suppresses noise cleverly will one day suppress the real page. Everything below is arranged around those two facts.

The platform is two planes. The paging path receives alerts, decides whether to page, reads who is on call, runs the escalation, speaks to carriers and accepts acknowledgements. It runs as three paging cells, one in each estate data centre and one in a colocation outpost that shares neither their power nor their network. Each cell is three dedicated hosts running static Go services under systemd, HAProxy at the edge, and NATS JetStream holding the paging state, with a site-local domain for bulk ingest and a paging domain with one replica per site. Asterisk and Jasmin speak SIP and SMPP to two independent carriers, and a push relay speaks to Apple and Google. The control plane edits schedules and policies, runs the response, holds the record and produces reviews. It runs on a small platform-owned Kubernetes cluster at one site with a warm rebuild at the other. Schedules are resolved there, 72 hours ahead, verified by a second independent resolver, and published into the paging state as 15-minute coverage buckets. The incident timeline is an append-only PostgreSQL log with a hash chain; the console reads projections of it. The notification ledger is written by the dispatcher and archived in ClickHouse, which also holds raw payloads and the noise and cost reports. Keycloak brokers the corporate identity provider and holds break-glass accounts; OpenBao holds every secret and the per-responder keys that protect contact data.

What it is, and what it is not

A paging path that reads a pre-resolved snapshota schedule database, a console or an identity provider consulted while a page is being decided.
Escalation that advances on a missing acknowledgementescalation that trusts a carrier's delivered receipt to mean a human saw the page.
Duplicate pages accepted as the price of never missing oneexactly-once delivery pursued at the cost of a page that was never sent.
Suppression declared by a named person, with an expiry, always recordeddependency-aware suppression that decides on its own that a second failure is the same failure.
An event log that is the record, with every view derived from ita mutable incident row whose history is whatever the last editor left.
Three sites, one of them outside the estate, from the first releasea single data centre with a disaster-recovery plan to add a second later.

The decisions that are the architecture

01The paging path is a separate plane

It holds a replicated copy of who is on call and how to escalate, and decides, dispatches and accepts acknowledgements with every other part of the platform unavailable. The rest of the design follows from this line.

ADR-01

02Nothing it pages for is underneath it

Paging cells run on dedicated hosts with no orchestrator, no shared DNS and no identity provider at run time, because the estate's Kubernetes, DNS and single sign-on are all things it must page about.

ADR-02

03Quorum governs the record, never the page

A cell cut off from the other two keeps paging from its last snapshot and labels what it did. Two pages for one problem is survivable; no page is not.

ADR-04

04Coverage is resolved ahead and verified twice

72 hours of 15-minute buckets, published within 60 seconds of a change, checked by an independent second resolver before paging can read them. A DST night is computed three days early.

ADR-14

05Only an acknowledgement stops escalation

Receipts move a page to another route or channel inside a step. They never advance a step and never end one.

ADR-16

06Carriers are aggregated, the last software hop is owned

Asterisk and Jasmin on two carriers with separate contracts and networks, and a responder app of the platform's own, so a keypress, an SMS reply and a button are acknowledgements the platform receives itself.

ADR-20

07The platform may decide not to page only in ways a human declared

Maintenance windows with an owner and an expiry, and flood control that attaches rather than drops. SEV1 and human-declared incidents are never suppressed.

ADR-12

08The log is the record

Every alert, page, acknowledgement, role change and correction is an appended event with a hash chain. The review's intervals are computed from it, not typed in.

ADR-08

Why it holds up over time

An incident platform outlives several generations of monitoring tools, chat products, carriers and phones. The design keeps the decisions that encode organisational judgement, who is on call, what counts as an acknowledgement, when the platform may stay silent, in data and in written rules, and keeps every product behind a seam narrow enough to replace without touching those rules.

The boundary is a rule, not a product

'The paging path must work with everything else down' can be tested against any future component in one question: does this add something the paging path waits for? It survives a change of broker, of language, of carrier and of data centre.

Carriers and chat are adapters

SIP, SMPP, APNs, FCM and SMTP are decades-old or platform-owned protocols. Changing carrier is a trunk and a route; changing chat product is one bridge. Neither touches escalation logic.

Time is handled once, and checked twice

Every stored instant is UTC, every rotation is a named IANA zone, and two independent resolvers must agree. Zone rules will keep changing; governments do not consult on-call rotas. The design makes that a routine release.

The record does not depend on today's schema

An append-only log of attributed events can be re-projected into whatever view a future console, analytics tool or regulator needs. A mutable record can only ever answer the questions its current columns anticipated.

The bias is written down

Duplicate over miss, acknowledgement over receipt, declared over inferred suppression. Future teams under pressure to reduce noise or cost will meet these as stated principles with their reasons, not as accidents of an old implementation.

Independence has a price on it

The outpost costs a known amount a year and is reviewed as insurance. A cost that is stated survives budget reviews; a cost that is discovered gets cut in the first one.

Fallbacks are exercised, so they stay real

Cell loss, control-plane rebuild, partition, carrier loss, exhausted escalation and the manual fallback kit are drilled on a calendar. An untested fallback quietly stops working as the system around it changes.

The platform measures what it costs people

Interrupts per responder and actionable page rate are reported every week. A platform that makes its own human cost visible keeps its users; one that does not is replaced by whatever they route around it with.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.

QualityTargetHow it is metView
Paging path availability ≥ 99.99% monthly Three cells in three locations, any one sufficient; no control-plane, identity or estate dependency; minority cells keep paging 26
On-call lookup API ≥ 99.99% monthly Served by the paging cells from the same coverage buckets the dispatcher reads 11
Control plane availability ≥ 99.9% monthly; RTO ≤ 15 min Platform-owned Kubernetes at site A, warm rebuild at site B from manifests and a synchronous PostgreSQL standby, drilled quarterly 10
Receipt to first notification p95 ≤ 15 s, p99 ≤ 30 s Every step local to one cell; one key read for coverage; reserved CPU for dispatch 12
Acknowledgement to cancel p99 ≤ 5 s Compare-and-set on incident state; timers and dispatcher check state before each fire and send 19
Escalation timer accuracy ± 5 s at p99 1-second wheel per shard over durable timer records; lease takeover in ≤ 3 s after an owner stops 19
Coverage staleness p99 ≤ 60 s after a change Affected rotation re-resolved and verified, buckets published with quorum; staleness alarmed per cell 17
Ingest throughput 2,000/s sustained, 20,000/s for 120 s per site Per-integration quotas at HAProxy; site-local ingest domain; flood control attaches as counts 21
Notifications 60,000/month; 6,000 in 10 min Asterisk and Jasmin per cell on two carrier routes, sized for the burst with one host down 20
Coverage snapshot durability RPO 0; paging RTO ≤ 2 min Quorum writes to one replica per site; anycast withdrawal moves traffic from a lost cell 14
Event log and ledger RPO ≤ 5 s Paging events in a 30-day three-site outbox before the log; PostgreSQL synchronous standby at site B 18
Projection rebuild ≤ 60 min, tested quarterly Projections dropped and replayed from the event log in the drill, open incidents first 18
Console incident list p95 ≤ 1,500 ms Read from the operational projection, never from the log 10
Retention Events and audit 7 y; ledger 13 mo; raw payloads 90 d PostgreSQL partitions with daily hash anchors in Ceph Object Lock; ClickHouse TTLs 14
Human cost ≤ 2 out-of-hours pages per responder per week; ≥ 70% actionable Measured from the ledger and incident events, reported weekly per rotation, never enforced by suppression 29

Scope

In scope

  • Alert ingest over webhook and email, normalisation, deduplication, quarantine and raw payload retention
  • Incident state machine, severity model, grouping, merge, split and human declaration
  • Layered schedules, overrides, timezone-correct resolution, coverage snapshots and gap detection
  • Escalation policies, push, SMS, voice and email across two carriers, and acknowledgement from every channel
  • Maintenance windows, flood control, noise reporting and every suppression decision recorded
  • Incident channels, roles, stakeholder updates and the service catalogue as the source of ownership
  • Post-incident reviews with computed intervals, action items synchronised with the tracker, and cross-incident search
  • The append-only event log, the notification ledger, the platform API and the outbound event stream

Explicitly out of scope

  • Metric evaluation and alert rule authoring, which stay in the monitoring systems
  • The chat product and the issue tracker themselves; the platform integrates with both
  • A customer-facing status page beyond the static page the outpost serves
  • Carrier networks and Apple and Google push services, which no on-premises design can own
  • Any managed cloud service; every component runs on hardware the organisation owns

Prove it before production depends on it

Eight weeks, three real locations, two carrier contracts on trial terms, and four volunteer rotations. Each item replaces an assumption in this record with a measurement.

  1. Measure JetStream key-value commit latency at p50 and p99 with one replica per site, under the storm replay, and with site C degraded to 30 ms
  2. Kill the owner of a timer shard mid-escalation and measure fire accuracy across 1,000 takeovers
  3. Send 10,000 SMS and 2,000 calls per carrier to the responders' real numbering ranges and measure receipt fidelity, DTMF recognition and time to ring
  4. Apply for the Critical Alerts entitlement in week one, and test the voice-bypass fallback on iOS and Android in parallel
  5. Partition site C for an hour during synthetic load and review every duplicate the reconciler labels
  6. Rebuild projections from a synthetic seven-year log and time it
  7. Resolve every rotation across every DST transition to 2030 with both resolvers and review every disagreement
  • Pass: synthetic pages from all three cells at p95 under 15 s for the final four weeks, including during the storm replay.
  • Pass: no timer fires more than 5 s late across the takeover test, and none fails to fire.
  • Fail and rethink: either carrier's receipts arrive for fewer than 90% of delivered test messages, which moves the silence threshold from receipts to acknowledgement time alone.

Open risks, carried rather than hidden

RiskIf it landsResponse
Duplicate pages during partitions and failovers A responder receives the same page from two cells, or a second call after acknowledging on the other side of a partition Accepted and stated as the deliberate bias. Every duplicate is labelled by the reconciler and counted; the count is a reported metric (ADR-04)
Critical Alerts entitlement refused SEV1 push on iOS cannot bypass do-not-disturb Voice from a number responders mark as an emergency bypass contact, configured at onboarding and verified by a test call (ADR-20)
Carrier receipts are unreliable Silence thresholds fire too often, producing unnecessary calls Thresholds are per carrier and per channel, measured continuously by loopback traffic; escalation never depends on receipts (ADR-16, ADR-21)
Built paging core becomes a bespoke burden A small team owns an escalation engine, timers and a dispatcher The built surface is five services with narrow contracts and a conformance suite; everything else is adopted. GoAlert is the named fallback if the team cannot sustain it (ADR-05)
Outpost treated as a cost to cut Losing site C removes the only cell outside the estate's failure domain Its premium is stated in the budget as insurance and reviewed yearly against the incidents it covered (ADR-32)
Stretched quorum latency to site C Paging-domain writes slow down when the outpost link degrades Writes need two of three replicas, so A and B commit without C; bulk ingest never crosses sites (ADR-07)

Architecture Decision Record

Why every component and every technology on these 34 views is what it is, and what each choice costs.

Thirty-two decisions make up this architecture. Everything else on the thirty-four views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the products, the team and the estate have changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Latencies, volumes and costs are the requirement's targets or planning estimates, and each is replaced by a measurement in the proof phase. Six behaviours are relied on and must be confirmed on the exact releases and contracts chosen before production depends on them: NATS JetStream key-value writes committing within 50 ms at p99 with one replica at each of three sites; JetStream's per-key TTL behaving as documented for the 24-hour deduplication and 60-second human-dedup buckets; both carriers returning SMS delivery receipts and passing DTMF reliably enough to measure, on the numbering ranges the responders use; Apple granting the Critical Alerts entitlement to the responder app; Asterisk and Jasmin sustaining the 6,000-notification, ten-minute major-incident burst on one cell with one host down; and PostgreSQL rebuilding the operational projections from seven years of events inside 60 minutes.

How to read a record

QuestionThe forcing question: why a decision was needed at all.
ContextThe requirement, the scale and the constraint that make it hard.
DecisionWhat this architecture does, stated so it can be checked.
How it works on-premiseThe concrete mechanism: which package, configured how, on whose hardware.
Options weighedChosen, rejected, deferred, or right elsewhere, with the reason for each.
ConsequencesWhat the choice buys and what it costs, both kept visible.
Choose differently whenThe conditions that would flip the decision for your system.
Why it holds up over timeWhat keeps the decision right as scale, staff and technology change.
LessonThe principle that transfers beyond this platform.

Decision map

Planes and independence 5

What sits on the path that wakes a human, what it is allowed to depend on, where each plane runs, and what happens when the sites cannot see each other.

ADR-01The paging path is a separate plane from the incident record ADR-02Paging cells run on dedicated hosts with no estate dependency at run time ADR-03Three paging cells active-active, one outside the estate; control plane single-site with a warm rebuild ADR-04Quorum governs the record, never the page ADR-05Build the paging core; adopt open source for everything around it

State and records 4

Where paging state lives, how bulk traffic is kept off the site links, what the system of record is, and where evidence and reporting data go.

ADR-06NATS JetStream holds the paging path's state ADR-07Two JetStream domains: site-local ingest, three-site paging ADR-08The incident timeline is an append-only PostgreSQL log; the record is a projection ADR-09ClickHouse for the ledger archive, raw payloads and reporting; Ceph Object Lock for tamper evidence

Ingest and decide 4

How untrusted alerts become normalised signals, who groups them, when the platform may stay silent, and what severity means.

ADR-10Ingest edge: per-integration quotas at HAProxy, verify before parse, CEL mappings, quarantine ADR-11Grouping is owned here, on explicit keys that never cross a service boundary ADR-12The platform may decide not to page only in ways a human declared, and always says why ADR-13Severity is declared, and it binds the response obligations

Schedules and escalation 5

Where on-call is resolved, how time zones are made boring, what escalation advances on, how timers survive a crash, and what a handoff does to a live page.

ADR-14On-call is resolved ahead of time into 15-minute coverage buckets, 72 hours out ADR-15Timezones are resolved in IANA zones, checked by a second resolver, and gaps are found 14 days out ADR-16Escalation advances only on absence of acknowledgement; delivery failure moves within a step ADR-17Escalation timers are sharded in-memory wheels over durable timer records, taken over by lease ADR-18A handoff adds the incoming responder to a live page; it never removes the outgoing one

Notification 5

Whether to own delivery or aggregate it, how providers fail over, what counts as a duplicate, how every channel acknowledges, and what happens when nobody does.

ADR-19Deduplicate at the human, not at the sender; prefer sending twice to not sending ADR-20Aggregate carriers, own the last software hop: Asterisk, Jasmin, a push relay and a responder app ADR-21Fail over on the same channel after a hard failure, to the next channel after silence ADR-22Every channel acknowledges with a signed, single-use token; no login, no DNS dependency ADR-23Escalation exhaustion is a terminal state, a repeating broadcast and an incident against the platform

Response and learning 2

How the response is coordinated around the incident record, and how a review is made the cheapest thing to finish.

ADR-24Coordination happens around the record: channels, roles, updates and ownership as events ADR-25Reviews are generated from the log, blameless by template, with action items tracked both ways

Security and identity 4

Signing in when the identity provider is down, integrations that are never trusted, contact data that can be erased from an immutable system, and incidents only some people may read.

ADR-26Keycloak brokers the corporate identity provider and holds break-glass accounts on hardware keys ADR-27Every integration is untrusted, independently authenticated and revocable without a deploy ADR-28Contact data is encrypted per responder, so offboarding erases it from an immutable system ADR-29Incidents are readable by default; sensitive ones are restricted by row-level security, and the restriction is audited

Operations and cost 3

How the platform proves it works from outside itself, how the paging path changes without becoming the outage, and how money and sleep are kept visible.

ADR-30A synthetic page runs every minute, an outside watchdog pages on carrier B, and a manual fallback kit is drilled ADR-31The paging path changes one cell at a time, never during a SEV1 or SEV2, from artefacts already on the host ADR-32Money and sleep are metered per service and rotation; the independence premium is a stated number

Technology by capability

Every capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns; carriers and Apple and Google push are the only external services, and none of them is a single dependency.

Open source This design
CapabilityChoiceOriginCredible alternativeWhy this oneRecord
Plane separation Paging path apart from the incident record This design One highly available application 99.99% paid for on the path that wakes people only ADR-01
Paging-path runtime Static Go binaries under systemd on dedicated hosts Open source Estate Kubernetes; per-cell Kubernetes Nothing it pages for is underneath it ADR-02
Site topology Three active cells, one outside the estate; warm control plane This design Two sites; active-active control plane Survives losing both estate sites; control plane gets the RTO asked for ADR-03
Partition behaviour Minority cells keep paging and label events This design Stop without quorum Duplicates are survivable; silence is not ADR-04
Build boundary Build paging core; adopt the rest This design GoAlert; Grafana OnCall OSS Requirement-specific core owned, general operations borrowed ADR-05
Paging state and streams NATS JetStream Open source Apache Kafka; etcd Streams, KV with CAS and TTL, Raft placement, one binary ADR-06
Storm isolation Site-local ingest domain, three-site paging domain This design One stretched domain Bulk traffic never crosses a site link ADR-07
System of record PostgreSQL append-only event log on CloudNativePG Open source Mutable record; dedicated event store Defensible history on a database the team can restore ADR-08
Ledger archive and reporting ClickHouse Open source PostgreSQL; OpenSearch Columnar scans and TTL retention for 13 months of attempts ADR-09
Tamper evidence Ceph RGW with Object Lock Open source WORM appliance Immutability enforced outside the database ADR-09
Ingest edge HAProxy stick-table quotas, Go gateway, CEL mappings Open source Per-source adapter code Cheap refusal of the offender; mappings without deploys ADR-10
Email ingest Postfix, per-integration addresses Open source IMAP polling Push delivery, sender allowlists, no mailbox dependency ADR-10
Grouping Explicit keys within a service This design Topology-aware correlation Never hides a second failure in another service ADR-11
Suppression Declared, expiring, recorded; attach not drop This design Automatic dependency suppression Silence is something a person signs for ADR-12
Severity Declared, bound to obligation profiles This design Computed from signals Humans correct the rule and obligations follow ADR-13
Coverage 72 h of 15-minute buckets in JetStream KV This design Page-time resolution One read, survives three days of control-plane outage ADR-14
Timezone correctness Two independent resolvers (Go and Python zoneinfo) This design One resolver with tests Disagreement catches what one author's tests miss ADR-15
Escalation trigger Step timer on missing acknowledgement This design Advance on delivery failure Receipts speed up a step; only acknowledgement ends one ADR-16
Timers Sharded wheels over durable KV records with leases This design Broker scheduled messages; polled rows Second-level accuracy, takeover in 3 s ADR-17
Handoff Union of outgoing and incoming until acknowledged This design Pin or re-resolve No unpaged minute at a boundary ADR-18
Duplicate control 60-second claim per responder, channel, incident, step This design Exactly-once dispatch Retries stay safe; uncertainty sends ADR-19
Voice Asterisk with PJSIP trunks to two carriers Open source FreeSWITCH IVR with DTMF acknowledgement, widely operated ADR-20
SMS Jasmin SMS gateway over SMPP to two carriers Open source Kannel SMPP routing with inbound replies as acknowledgements ADR-20
Push Push relay to APNs and FCM; own responder app This design Self-hosted UnifiedPush relay Only route that wakes iOS; entitlement held by own app ADR-20
Provider failover Same channel on failure, next channel on silence This design Same channel on any trigger Fast failover without duplicate SMS ADR-21
Acknowledgement Signed single-use tokens on every channel This design Console login Works through identity and DNS outages ADR-22
Exhaustion Terminal state, repeating broadcast, platform SEV2 This design Silent close The worst outcome is the loudest ADR-23
Chat Mattermost Team Edition via a chat bridge Open source Matrix with Element; Zulip Existing organisational chat; asynchronous, never on the paging path ADR-24
Service catalogue YAML in Git, Backstage catalog-info format This design Backstage as runtime One reviewed ownership source, readable by other tools ADR-24
Reviews and search Generated drafts; PostgreSQL full-text search Open source Wiki; OpenSearch Facts computed, blameless by template, search at the right scale ADR-25
Identity Keycloak broker with WebAuthn break-glass Open source Direct federation Console reachable during identity outages, with fewer powers ADR-26
Secrets OpenBao Open source HashiCorp Vault Vault's API under an open-source licence (MPL 2.0) ADR-27
Contact data protection Per-responder keys via OpenBao transit This design Row deletion Erasure across immutable stores and backups ADR-28
Sensitive incidents PostgreSQL row-level security, audited restriction Open source Separate tool Enforced for every reader; nothing hidden quietly ADR-29
Self-verification Minute synthetic, site-C dead-man switch, Gammu fallback kit This design Internal health checks Watched from outside, fallback actually exercised ADR-30
Release Forgejo, Harbor, cosign, Ansible in gated cell waves Open source All cells at once One failure domain at a time, rollback without network ADR-31
Cost and human load Ledger-priced attempts, ClickHouse weekly report This design Central unattributed budget Noise has a price its owner sees ADR-32

The decisions, and the alternatives that lost

Planes and independenceWhat sits on the path that wakes a human, what it is allowed to depend on, where each plane runs, and what happens when the sites cannot see each other.

ADR-01

The paging path is a separate plane from the incident record

Accepted

What must still work for a human to be woken when the rest of the platform is unavailable?

Context
The platform is used when other systems are failing and often when its own dependencies are part of the failure. The requirement asks for 99.99% on the path from alert to acknowledgement and 99.9% for the console, schedule editing and reviews. A single application with one database gives every capability the availability of its least reliable part, and the least reliable parts are the richest ones: the console, search, reviews and analytics.
Decision
Split the platform into two planes. The paging path ingests, decides, reads coverage, runs escalation, dispatches and accepts acknowledgements, using only a replicated copy of the two things it needs: who is on call, and how to escalate. The control plane edits schedules and policies, runs the response, holds the event log and produces reviews. The paging path publishes events to an outbox that the control plane reconciles later. Nothing in the paging path calls the control plane.
How it works on-premise
Paging services run on dedicated hosts in three paging cells (ADR-02, ADR-03) with NATS JetStream as their only shared state (ADR-06). The control plane runs on a platform-owned Kubernetes cluster. The only data flowing from control plane to paging path is coverage buckets and escalation policies written into JetStream key-value buckets; the only data flowing back is the outbox. A conformance test fails any paging-path build that opens a connection to a control-plane address.
Options weighed
  • ChosenTwo planes with a replicated snapshot: The paging target is met at the cost of the paging path alone, and the control plane can be built, changed and restored at ordinary cost.
  • RejectedOne application, made highly available: Simpler to build and reason about. Every feature added to the console becomes a way to break paging, and 99.99% has to be paid for across all of it.
  • RejectedOne application with a degraded read-only mode: Keeps one codebase. The degraded mode is exercised only during outages, which is when it is least likely to have kept up with the code around it.
  • Right elsewhereBuy a SaaS paging service in front of an on-premises record: Right for an organisation allowed to use one, and the independence argument is good. Excluded here by the open-source, fully on-premises constraint.
Consequences
What it buys
  • The console, schedule editor, reviews and analytics can all be down with no effect on paging
  • Each plane gets the engineering discipline its target deserves and no more
  • The critical surface is small enough to test exhaustively, including its failure modes
What it costs
  • Two representations of on-call state and a staleness window between them
  • A class of bug where the page goes to whoever was on call a minute ago
  • Reconciliation of paging events into the record is a component that must itself be correct
Choose differently when
When the organisation is small enough that one rotation covers everything and a person can be phoned by hand, a single well-run application with a tested manual fallback costs less and fails in simpler ways.
Why it holds up over time
This is a dependency rule, not a technology choice. The broker, the language and the carriers can all change and the rule still decides every design review: a proposal either adds something the paging path waits for, or it does not.
LessonGive the part that must not fail the smallest possible set of things it needs, a copy of each, and no way to ask for more.
Shown on views02 08 22
ADR-02

Paging cells run on dedicated hosts with no estate dependency at run time

Accepted

What infrastructure may the paging path stand on, given that it must page about that infrastructure failing?

Context
The organisation's Kubernetes clusters, DNS, identity provider, container registry, secrets service and network core are all things the platform will page about. Each is a reasonable dependency for an ordinary service and a correlated failure for this one. A paging service scheduled on the estate's Kubernetes cannot restart when the scheduler is the incident; one that resolves carrier hostnames through corporate DNS cannot page when DNS is the incident.
Decision
Run the paging services as static binaries under systemd on hosts dedicated to the paging path, three per cell. At run time they need only their local disk, the other cells over mTLS, carrier endpoints by IP address, and time from the cell's own sources. Artefacts, configuration and tzdata are staged on every host before a release switches to them.
How it works on-premise
Go services built with CGO disabled, installed by Ansible from signed OCI artefacts pulled from Harbor ahead of the switch and kept locally. HAProxy fronts each cell with an anycast address announced by BIRD over BGP with BFD. Each cell runs local caching resolvers with carrier and Apple and Google endpoints pinned, and chrony against a GPS-disciplined source at the outpost and two independent public pools elsewhere. Hosts are 16-core, 64 GB, 2 TB NVMe machines with disk encryption bound to the TPM.
Options weighed
  • ChosenDedicated hosts, systemd, staged artefacts: Nothing the estate runs is underneath the paging path. A small, boring operational model with few moving parts.
  • RejectedThe estate's shared Kubernetes clusters: Free to operate and familiar. A control-plane or CNI incident in the estate removes the ability to restart or reschedule the thing paging about it.
  • RejectedA small dedicated Kubernetes cluster per cell: Independent of the estate. Adds etcd, a CNI and an API server to a path that runs five processes, and a cluster upgrade becomes a paging-path change.
  • RejectedNomad per cell: Lighter than Kubernetes and fine technically. Relicensed under the Business Source License in 2023, which fails the open-source constraint.
Consequences
What it buys
  • A failure of the estate's orchestration, DNS, registry or identity cannot stop paging
  • Restart and rollback are local operations: a symlink and a systemd restart
  • The host inventory is small enough to rebuild from Ansible in under an hour
What it costs
  • Nine hosts managed outside the organisation's standard platform, with their own patching
  • No automatic rescheduling: a lost host stays lost until replaced, covered by the other two in its cell
  • The platform team needs host-level operational skills the rest of the organisation may have handed to Kubernetes
Choose differently when
When the estate's orchestration runs in a failure domain genuinely separate from every service it hosts, and has been measured to be so through several major incidents, running the paging path there removes nine hosts from the platform team's care.
Why it holds up over time
The rule is 'the paging path does not stand on what it pages for', and it outlives any orchestrator. If the estate later moves to something other than Kubernetes, or the paging path moves to different hosts, the same test decides where it may run.
LessonA dependency that is reliable enough for everything else can still be exactly the wrong dependency for the system that reports its failures.
Shown on views09 26 27
ADR-03

Three paging cells active-active, one outside the estate; control plane single-site with a warm rebuild

Accepted

Where does each plane run, and how much redundancy does each one buy?

Context
The requirement names a correlated failure class: the region hosting both the estate and the platform. On-premises, the estate runs in two data centres, A and B, that share a network core, an identity provider and often an operations team. The paging path must survive losing both. The control plane has a 15-minute RTO and may be single-region. The requirement also says a hot second control plane buys nothing it asks for.
Decision
Run a paging cell in site A, one in site B and one in site C, a colocation outpost on a different power grid, upstream carrier and network path from A and B. All three cells accept alerts and page. Run the control plane at site A with a synchronous PostgreSQL standby and scaled-to-zero manifests at site B, rebuilt there within 15 minutes when A is lost.
How it works on-premise
Site C holds a full paging cell, carrier B's SIP trunk and SMPP bind, a GSM modem bank, the synthetic probes, the dead-man switch and the static status page. It connects to A and B over the organisation's WAN and, independently, over its own internet uplink with an IPsec tunnel, so losing the WAN does not isolate it. The control plane's site-B rebuild is an Argo CD sync of the same manifests plus promotion of the CloudNativePG replica, timed quarterly.
Options weighed
  • ChosenThree active cells, one outside the estate; warm control plane: Paging survives losing both estate sites; the control plane gets exactly the recovery the requirement asks for.
  • RejectedTwo cells in A and B only: Cheaper by the whole outpost. Shares every failure A and B share, which is precisely the correlated class the requirement names.
  • RejectedActive-active control plane in A and B: Removes the 15-minute rebuild. Doubles consistency work for the event log and the schedule store to buy an RTO the requirement does not ask for.
  • RejectedOutpost as a cold standby: Cheaper to run. A cold paging cell that has not handled real traffic is exactly the untested fallback the principles forbid.
Consequences
What it buys
  • The event that takes out both estate data centres is covered by a cell that has been paging all along
  • Site C is also the natural home for everything that must observe the platform from outside
  • The control plane stays simple enough to restore by a small team under pressure
What it costs
  • A stated yearly premium for site C (ADR-32)
  • Three cells to keep in step through releases, ordered and gated (ADR-31)
  • The console is unavailable for up to 15 minutes if site A is lost, with cell status pages in the meantime
Choose differently when
An organisation whose two data centres are genuinely independent, different regions, carriers, power and operations teams, with no shared network core or identity, can treat B as the outside cell and save the outpost.
Why it holds up over time
The shape, some cells inside the estate and at least one outside it, holds whatever the estate becomes. If the estate moves to a cloud, the outside cell stays on-premises; if it stays on-premises, the outside cell can move to a different provider. The rule is that one cell never shares the estate's fate.
LessonRedundancy only counts against the failures it does not share. Count the shared failures before counting the copies.
Shown on views09 10 26
ADR-04

Quorum governs the record, never the page

Accepted

What does a paging cell do when it cannot reach a majority of the paging domain?

Context
The paging domain uses Raft with one replica per site. A cell partitioned from the other two cannot commit writes: it cannot open an incident with a compare-and-set, write a timer record or record an acknowledgement durably. The conventional answer for a replicated system is to stop writing and serve nothing. For this system the partitioned cell may be the only one that can still see a failing estate and still reach a carrier.
Decision
A cell that loses quorum enters minority mode. It continues to decide and page using its last coverage snapshot and policies, keeps escalation state and events in a local append-only log, accepts acknowledgements it can receive, marks every message it sends as coming from a partitioned cell, and pages the platform rotation about its own state. When quorum returns, the local log is replayed through the reconciler, which links and labels duplicate incidents rather than merging them.
How it works on-premise
Each paging service watches its JetStream connection's cluster state and the paging domain's meta-leader. After 10 seconds without quorum it switches to minority mode, writing to a local file-backed log on NVMe with fsync per event. Coverage buckets are read from the local replica, which JetStream keeps serving for direct gets. The reconciler replays minority logs idempotently by event id and writes partition-duplicate links as events.
Options weighed
  • ChosenMinority cells keep paging and label what they did: No page is lost to a partition. The cost is duplicates, which the design already accepts and counts.
  • RejectedStop paging without quorum: The textbook answer for consistency. Guarantees that the outpost is silent in exactly the event it exists to cover.
  • RejectedLeaderless design with CRDT state: Avoids the mode switch entirely. Much harder to reason about for acknowledgements and timers, and no mature open-source store fits the paging state's shape.
  • RejectedManual switch to minority mode by an operator: Keeps humans in charge of a risky mode. Needs a human awake and aware during the event that is waking nobody.
Consequences
What it buys
  • Every partition shape leaves at least one cell able to page whoever it can reach
  • The bias toward duplicates is implemented, not just stated
  • The review can see exactly what each side of a partition did
What it costs
  • A responder may be paged by both sides of a partition, and called again after acknowledging on the other side
  • Minority mode is a second code path and must be exercised by the quarterly partition drill
  • The reconciler must handle incidents that exist twice for a while
Choose differently when
If duplicate pages carried real harm, for example a page that dispatches a physical crew, the balance changes and a minority cell should page only a small, explicitly configured set of escalation targets.
Why it holds up over time
The principle is independent of Raft or JetStream: consistency protocols decide what a system may record, and a paging system must not let them decide whether it speaks. Any future replacement for the paging state inherits the same split.
LessonDecide separately what your consistency model protects and what it is allowed to prevent. They are not always the same thing.
Shown on views12 23 34
ADR-05

Build the paging core; adopt open source for everything around it

Accepted

Is there an open-source incident platform to adopt, or does the core need to be built?

Context
The requirement is unusually specific about the paging path: a separate plane, ahead-of-time coverage, acknowledgement-driven escalation with durable timers, provider failover inside a step, human-side deduplication, minority operation and an event-sourced record. Open-source options exist. GoAlert, from Target, is mature and well designed for its purpose, with PostgreSQL as its single store and Twilio for SMS and voice. Grafana OnCall's open-source edition was put into maintenance mode in 2025 with archival announced. Netflix Dispatch coordinates response and leaves paging to an external on-call service.
Decision
Build five paging services (ingest gateway, incident engine, escalation timers, dispatcher, ack receiver) and the control-plane domain services (schedule resolver, incident service, review module, console). Adopt open source for every other part: edge, messaging, databases, voice, SMS, identity, secrets, chat, observability and delivery. Keep GoAlert as the named fallback if the team cannot sustain the built core.
How it works on-premise
The built services are Go, with a TypeScript console. Each paging service has a written contract, a property-based test suite and a conformance suite run against every release: timer accuracy across restarts, acknowledgement cancellation latency, human-dedup behaviour under failover, and minority mode. The built surface is sized at roughly 30,000 lines for the paging path, owned by a team of six.
Options weighed
  • ChosenBuild the core, adopt the rest: The parts that encode the requirement are owned; the parts that encode general operations are borrowed from projects with far more production mileage.
  • Right elsewhereGoAlert: The right answer for an organisation that can use Twilio and accept PostgreSQL on the paging path. Page-time schedule resolution and a single database are exactly what this requirement separates out.
  • RejectedGrafana OnCall open-source edition: Feature-rich and familiar to Grafana users. Its open-source edition is in maintenance mode, which is the wrong starting point for a ten-year platform.
  • RejectedNetflix Dispatch: Strong on response coordination and reviews. It does not page; it would still need everything this design builds on the paging path.
Consequences
What it buys
  • The paging path is exactly what the requirement asks for, with no inherited assumption to work around
  • Every adopted component has a large user base and published failure behaviour
  • The team's engineering effort goes to the part that differentiates reliability
What it costs
  • A team must own an escalation engine, timers and a dispatcher indefinitely
  • Bugs in the built core have no upstream community to find them first
  • Feature requests common in commercial tools, such as rich mobile apps, arrive slowly
Choose differently when
An organisation of a few dozen responders on one site, allowed to use a cloud SMS and voice provider, should run GoAlert. Most of this design's complexity pays for independence and scale that organisation does not need.
Why it holds up over time
The build boundary sits at the requirement, not at a product. If a future open-source project implements a separate paging plane with ahead-of-time coverage, the built core can be retired behind the same event and coverage contracts without touching the control plane or the record.
LessonBuild the part whose requirements you would have to fight a product to meet, and nothing else.
Shown on views08 30

State and recordsWhere paging state lives, how bulk traffic is kept off the site links, what the system of record is, and where evidence and reporting data go.

ADR-06

NATS JetStream holds the paging path's state

Accepted

What durable, replicated store and log does the paging path share across cells?

Context
The paging path needs a small amount of strongly consistent state (open incidents by grouping key, escalation state, timer records, acknowledgements, coverage buckets, policies), a durable ordered log for the outbox and the ledger, key-value entries with per-key expiry for deduplication, and a site-local buffer for bulk ingest. It must replicate across three sites, run on nine hosts without an orchestrator, and be operable at 03:00 by the same team that runs the rest of the path.
Decision
Use NATS JetStream for all of it: streams for ingest, outbox and ledger; key-value buckets for coverage, policies, incident state, timers and deduplication. One binary, one operational model, one client library in every paging service.
How it works on-premise
nats-server on every paging host, configured as two JetStream domains (ADR-07). Key-value buckets use three replicas with one per site; streams for ingest use three replicas within a cell. Per-key TTL is used for the 24-hour alert dedup and the 60-second human dedup. Accounts separate ingest, paging and control-plane publishers, and every connection uses mTLS from the platform PKI.
Options weighed
  • ChosenNATS JetStream: Streams, key-value with compare-and-set and TTL, Raft replication with placement, a single small binary. A CNCF project under Apache 2.0.
  • RejectedApache Kafka: The strongest choice for high-volume logs and long retention. No key-value or per-key TTL primitive, heavier to stretch across three sites, and more to operate than a five-process path needs.
  • RejectedPostgreSQL with synchronous replication: Familiar and transactional. Puts a relational database with failover semantics in the paging path, and does not provide a site-local buffer for storms.
  • Rejectedetcd: Excellent small consistent store. No streams, a small value size limit, and not designed for the ingest or ledger volumes.
  • RejectedRedpanda: Kafka-compatible and fast. Business Source License, which fails the open-source constraint.
Consequences
What it buys
  • All paging state has one replication and failure model to understand and drill
  • Compare-and-set on grouping keys and incident state gives exactly-one incident while sites can talk
  • The footprint is small enough for three hosts per cell to carry comfortably
What it costs
  • The team must learn JetStream's Raft behaviour, placement and recovery in depth
  • Long-retention analytics cannot live in JetStream and are sunk to ClickHouse
  • A defect in JetStream affects the whole paging path at once, which is mitigated only by staged upgrades one cell at a time
Choose differently when
If the paging path grew to handle millions of events a second with long retention, or the organisation already ran Kafka as a mature internal service outside the estate's failure domain, Kafka for streams plus a separate consistent store would be the better split.
Why it holds up over time
The paging services depend on four operations: publish to a stream, consume from a stream, compare-and-set a key, get a key. Any future store offering those four with replication can replace JetStream behind a thin internal interface, and the contract tests already exercise exactly those four.
LessonWhen a critical path needs a few kinds of state, one well-understood store that provides all of them beats the best store for each.
Shown on views08 09 14
ADR-07

Two JetStream domains: site-local ingest, three-site paging

Accepted

How is a 20,000-alert-a-second storm kept from competing with paging state for the links between sites?

Context
Replicating every alert across three sites would push storm traffic, 2.4 million messages with payloads, through the Raft groups that also carry acknowledgements and timer writes. Site C's link is the slowest and least predictable. Paging state writes are few, a handful per incident, and must be replicated across sites. Alert writes are many and need durability within a site only, because an alert is reproducible by its source until acknowledged with a 202.
Decision
Run two JetStream domains on the same hosts. The ingest domain is local to each cell, with three replicas across that cell's hosts, and carries raw and normalised alerts. The paging domain has one replica per site and carries incident state, timers, acknowledgements, coverage, policies, the outbox and the ledger. Only the incident engine crosses from one to the other, and only with decisions.
How it works on-premise
Separate nats-server processes and ports per domain on each host, with separate storage directories on the NVMe and separate CPU weights in systemd. Ingest streams are limited to 7 days or 400 GB per cell. The paging domain uses placement tags so each replica lands in a different site. Commit latency on the paging domain is measured continuously and alarmed at 50 ms p99.
Options weighed
  • ChosenSite-local ingest, three-site paging: Bulk traffic never crosses a site link; paging state is replicated where it matters.
  • RejectedOne stretched domain for everything: One topology to operate. A storm becomes a cross-site replication problem at the moment acknowledgements most need low commit latency.
  • RejectedNo ingest buffer; engine consumes from HTTP directly: Fewer moving parts. The first slow decision backs up into sources and turns a storm into dropped webhooks.
Consequences
What it buys
  • Storm volume is bounded by one cell's disks and CPU, not by the WAN
  • Paging domain commit latency stays predictable during storms
  • A cell can be drained of ingest without touching paging state
What it costs
  • Alerts admitted by a cell that is then lost before the engine consumes them must be re-sent by their source, which the 202 contract makes explicit
  • Two domains per host to configure, monitor and upgrade
  • Deduplication is site-local, so a retry that lands on another cell attaches as a counted duplicate rather than being dropped
Choose differently when
If sites were connected by low-latency, high-capacity links with no meaningful cost or congestion, one stretched domain would be simpler and would make deduplication global.
Why it holds up over time
The separation is by the nature of the data, reproducible bulk signals versus irreplaceable decisions, and would hold on any messaging technology. It is also the seam along which ingest could move to a different store without touching paging state.
LessonReplicate decisions widely and signals locally. They have different volumes, different value and different owners of the retry.
Shown on views09 16 21
ADR-08

The incident timeline is an append-only PostgreSQL log; the record is a projection

Accepted

Is the incident record the source of truth, or a projection of an event log?

Context
The review, the audit obligation and the merge and split operations all need to answer questions about the past: what was believed at 03:14, who changed severity and why, what the incident looked like before two were merged. A mutable incident row answers only the present. Retention is seven years for timelines and audit events, which must be immutable. Volume is modest: 5,000 incidents a month and on the order of tens of millions of events a year including attached alert summaries.
Decision
Store every change as an appended event in PostgreSQL: alerts attached, state transitions, severity changes, pages, acknowledgements, role changes, notes, merges, splits, corrections. Each event has an idempotent id, a per-incident sequence, the platform receive time, an actor and the hash of the previous event in that incident. Incident views are projections rebuilt from the log. Corrections are new events that supersede earlier ones.
How it works on-premise
PostgreSQL 17 on CloudNativePG, primary at site A, synchronous standby at site B, one asynchronous replica for reporting. The event table is partitioned monthly. Application roles have INSERT and SELECT only, and a trigger rejects UPDATE and DELETE for every role including the owner. A daily job writes each incident's chain head and a partition checksum to a Ceph RGW bucket with Object Lock in compliance mode. Closed partitions older than two years are exported to Parquet in the same bucket and kept queryable.
Options weighed
  • ChosenAppend-only PostgreSQL log with a hash chain: A defensible record on a database the team already knows how to back up, restore and query.
  • RejectedMutable incident record with an audit table: Simpler to query and build. The audit table becomes the real record, maintained by hand, and merge and split destroy history.
  • RejectedA dedicated event store: Purpose-built stream semantics. One more database to operate and restore under pressure, for a volume PostgreSQL handles routinely.
  • RejectedKafka as the system of record: Natural log semantics. Seven years of immutable, queryable retention is not what Kafka is operated for, and projections would still need a database.
Consequences
What it buys
  • The review's intervals and responders are computed from facts, not typed from memory
  • Merge, split and correction preserve every earlier state
  • Tampering with history is detectable from outside the database
What it costs
  • Projection rebuild is an operational concern that must be drilled
  • Queries about current state go through projections, which can lag the log
  • Schema evolution of event payloads needs versioned readers rather than migrations
Choose differently when
For a platform that never merges or corrects incidents and has no audit obligation, a mutable record with a change log is cheaper to build and to query, and loses little.
Why it holds up over time
An attributed, ordered, append-only log can be re-projected into any future console, analytics model or regulator's question. The storage engine can change; the event contract, id, sequence, time, actor, type, payload, previous hash, is what carries seven years of history forward.
LessonIf the questions you will be asked are about the past, store the past. A current state can always be derived; a lost history cannot.
Shown on views14 15 18
ADR-09

ClickHouse for the ledger archive, raw payloads and reporting; Ceph Object Lock for tamper evidence

Accepted

Where do high-volume evidence and reporting data live once they leave the paging path?

Context
The notification ledger must be retained for 13 months and queried for cost, delivery fidelity and review evidence. Raw alert payloads must be kept for 90 days and searched when a mapping is wrong. The weekly noise and cost report aggregates alerts, incidents and attempts per service. These are append-heavy, scan-heavy workloads that would crowd the event log's database, and none of them may slow down the console.
Decision
Sink the ledger stream, raw payloads and alert metadata from JetStream into ClickHouse with table TTLs matching each retention. Build the noise, cost and interrupt reports as materialised views there. Keep hash-chain anchors and archived event-log partitions in a Ceph RGW bucket with Object Lock.
How it works on-premise
ClickHouse with two replicas at site A and one at site B, coordinated by ClickHouse Keeper, fed by consumers that commit JetStream acknowledgements only after a successful insert. Payloads use ZSTD compression. Grafana reads ClickHouse for dashboards; the weekly report is rendered by a job and emailed. Ceph RGW runs on the organisation's existing Ceph cluster with a dedicated pool and a compliance-mode Object Lock retention of seven years.
Options weighed
  • ChosenClickHouse plus Ceph Object Lock: Columnar storage built for exactly these scans, with TTLs that implement retention, and immutability enforced by the object store.
  • RejectedEverything in PostgreSQL: One database to run. Hundreds of millions of ledger and payload rows compete with the event log for I/O and vacuum.
  • RejectedOpenSearch: Excellent for free-text search over payloads. Weaker and more expensive for the aggregations that are most of the workload.
  • DeferredObject storage and a query engine such as Trino: Cheapest at very long retention. Adds a query engine to operate for data that is only kept 13 months.
Consequences
What it buys
  • Cost per page, per incident and per service is a query that returns in seconds
  • A wrong mapping can be diagnosed and re-run against 90 days of raw payloads
  • The event log's database carries only the record
What it costs
  • One more database technology for the team
  • Reports can lag the ledger by the sink interval, which the degradation order allows
  • ClickHouse's replication and merges need their own monitoring
Choose differently when
If payload search became the dominant workload, with free-text queries across months of vendor emails, OpenSearch for payloads beside ClickHouse for aggregates would be worth its extra footprint.
Why it holds up over time
The data leaves the paging path as streams with stable schemas. Any future analytical store can be fed from the same streams, and the Object Lock archive in Parquet is readable by most analytical engines that will exist in ten years.
LessonKeep the record's database for the record. Evidence and reporting grow faster and are queried differently.
Shown on views14 16 29

Ingest and decideHow untrusted alerts become normalised signals, who groups them, when the platform may stay silent, and what severity means.

ADR-10

Ingest edge: per-integration quotas at HAProxy, verify before parse, CEL mappings, quarantine

Accepted

How are heterogeneous, untrusted alert sources admitted and normalised without one of them degrading the others?

Context
Alerts arrive from metric and log alerting, synthetics, vendor health feeds, ticketing systems and email. Integrations will send malformed payloads, retry aggressively, flood during their own incidents and occasionally be hostile. The requirement asks for one normalised schema, the raw payload kept, per-integration quotas that shed only the offender, and a quarantine path that raises an incident against the integration owner rather than failing silently.
Decision
Admit at HAProxy with a per-integration rate limit keyed by credential. In the gateway, verify the HMAC signature and replay window before parsing, then evaluate the integration's CEL mapping in a sandbox with time and memory limits. Valid alerts are published with the raw payload in one message; invalid ones are published to quarantine with the reason. A quarantine monitor opens a SEV4 incident against the integration owner when its quarantine rate crosses a threshold.
How it works on-premise
HAProxy stick tables track request rate per integration id extracted from a header, returning 429 with Retry-After above the quota. The gateway is Go, using cel-go with a cost limit per evaluation. Mappings are versioned objects edited in the console by integration owners, published to the paging domain like coverage. Postfix accepts mail only for per-integration addresses from allowlisted senders and hands parsed messages to the same gateway path.
Options weighed
  • ChosenQuotas at the edge, CEL mappings, quarantine: Cheap refusal of the offender, mappings changeable without a deploy, and nothing silently rejected.
  • RejectedIntegration-specific adapter code per source: Full flexibility. Every new source is a paging-path release, and a bug in one adapter is a crash in the shared gateway.
  • RejectedRequire sources to send the normalised schema: No mapping layer at all. Vendors and existing tools will not change their payloads, so the adapter work moves to someone less able to test it.
  • RejectedReject invalid payloads with 400: Clean API semantics. A sender told 400 stops retrying, and the one real alert in a malformed payload is lost.
Consequences
What it buys
  • One noisy or broken integration costs only itself
  • Integration owners fix their own mappings, and can re-normalise past payloads
  • Quarantine volume is visible as an incident, not a log line
What it costs
  • CEL is one more language for integration owners to learn
  • Quotas need tuning per integration and review after each storm
  • Accepting malformed payloads with 202 means a broken integration can go unnoticed until the quarantine threshold
Choose differently when
A small, fixed set of sources all owned by one team can skip mappings and quotas and agree a schema directly; the machinery pays for itself when integrations are many and owned by different people.
Why it holds up over time
The normalised alert schema and the quarantine contract are the stable part. Sources, transports and even the expression language behind mappings can change without affecting anything after the ingest domain.
LessonRefuse cheaply, before understanding. Keep what you could not understand, and tell its owner.
Shown on views12 16 21
ADR-11

Grouping is owned here, on explicit keys that never cross a service boundary

Accepted

Who owns grouping: the monitoring systems, or this platform?

Context
Upstream alerting knows topology and intent; Alertmanager already groups by labels. This platform sees every source at once but does not own a dependency model. The organisation's worse error is the subtle one: a second, unrelated failure attached to an incident someone has already acknowledged, so nobody is paged for it. An alert storm that buries the signal is loud and gets fixed; over-grouping is quiet.
Decision
Group on an explicit grouping key: by default service, environment and the source's own group key when it provides one, otherwise the fingerprint family. Honour upstream grouping as an input, never replace it. An alert for a different service never joins an open incident automatically. An attached alert with a higher proposed severity, or from a service not yet in the incident, re-pages. Responders can merge, split and detach, as events that preserve both timelines.
How it works on-premise
The grouping key is computed in the incident engine from the normalised alert and the service catalogue entry, and claimed with a compare-and-set in the paging domain's open-incidents bucket. Alertmanager's groupKey passes through normalisation as source_group. Merge and split are control-plane commands that append events and update the open-incidents bucket through the same compare-and-set.
Options weighed
  • ChosenExplicit keys here, within a service, upstream honoured: Predictable grouping that a responder can explain, with a hard stop at service boundaries where hidden failures live.
  • RejectedLeave grouping entirely to upstream: Uses the richest knowledge. Different sources group differently, and a vendor email and a Prometheus alert for the same outage become two pages.
  • DeferredTopology-aware grouping across services: Collapses cascades into one incident. Needs an accurate dependency model and hides the second failure when that model is wrong; Phase 3, as a suggestion only.
  • RejectedTime-window grouping across all alerts: Easy to build and effective in storms. The canonical way to hide an unrelated failure that happens to start in the same minute.
Consequences
What it buys
  • A responder can always explain why two alerts are in one incident
  • A cascade across services produces one page per affected service, not one per alert, and not one in total
  • History survives every regrouping a responder performs
What it costs
  • Cascading failures page several rotations, each of which may see the same root cause
  • Grouping quality depends on the service catalogue being right
  • Merge and split are manual work during the worst incidents
Choose differently when
In an organisation where storms routinely page a dozen rotations for one upstream cause and the dependency graph is generated from real traffic rather than declared, topology-aware grouping earns its risk.
Why it holds up over time
The key, service boundary rule and re-page conditions are policy, stored as data. Better correlation can be added later as annotation without changing what is allowed to suppress a page, which is the part that must not drift.
LessonChoose the grouping error you can see. A storm is obvious; a hidden second failure is not.
Shown on views05 12 13
ADR-12

The platform may decide not to page only in ways a human declared, and always says why

Accepted

Where may suppression be automatic rather than declared in advance by a person?

Context
Maintenance windows, flood control and dependency-aware suppression all reduce noise, and each is a mechanism for missing a real incident. Paging for everything destroys the rotation and then trust. The requirement caps suppression at 30 days, requires owner and expiry, forbids suppressing SEV1 or human-declared incidents, and requires every suppression decision to be logged with its rule.
Decision
Evaluate five checks in a fixed order: exact duplicate, exemption (SEV1 or human-declared), declared maintenance window, attachment to an open incident, and flood control per grouping key. The only automatic silences are exact duplicates and attachment, and both record the alert. Maintenance windows must name an owner and expire within 30 days, renewed explicitly. Flood control attaches and counts; it never drops. Dependency knowledge annotates incidents and never withholds a page. Every decision writes an event naming the rule.
How it works on-premise
Suppression rules are control-plane objects, restricted to a narrower role than schedule editing, published into the paging domain as versioned policy. The engine evaluates them in memory against the published version and stamps the version on every decision event. A daily job lists rules expiring within seven days to their owners; expired rules stop matching at their expiry instant with no grace period.
Options weighed
  • ChosenDeclared, expiring, recorded; attach rather than drop: Noise is reduced by mechanisms a person chose and can be held to, and nothing is lost.
  • RejectedAutomatic dependency-aware suppression: The biggest single noise reduction available. Its failure mode is silence during a second, unrelated incident, which is the worst thing this platform can do.
  • RejectedMachine-learned noise suppression: Can find patterns humans miss. Cannot explain its decision in a review, and its errors are silent by construction.
  • RejectedPage for everything, rely on grouping: No suppression risk. Maintenance windows exist because planned work generates real alerts, and paging for them trains people to ignore pages.
Consequences
What it buys
  • A review can establish in one query whether the platform chose not to page, and on whose rule
  • No suppression outlives the memory of why it was created
  • SEV1 and human-declared incidents are immune to every configuration mistake
What it costs
  • More pages during cascading failures than a clever suppressor would send
  • Owners must renew long maintenance periods monthly
  • Noise reduction depends on people fixing alert rules, which the weekly report pushes but cannot force
Choose differently when
Where alert volume from cascades is so high that people are demonstrably ignoring pages, measured by acknowledge-and-resolve-with-no-action rates, a narrowly scoped automatic suppression with a strict audit may be the lesser harm.
Why it holds up over time
The ordering and the rule 'annotate, never withhold' are written principles, not code accidents. Future correlation technology can be as clever as it likes on the annotation side without being allowed to change who is woken.
LessonA suppressed real page costs more trust than a hundred false ones. Make silence something a person signs for.
Shown on views13 21 34
ADR-13

Severity is declared, and it binds the response obligations

Accepted

Is severity a computed property of alerts or a declared property of an incident, and what does it control?

Context
Alerting rules propose a severity, and they are often wrong in both directions. Responders learn more in the first minutes than any rule knew. The requirement asks for four levels, a proposed default, any responder able to raise or lower it with an attributed reason, and obligations bound to severity: acknowledgement target, channel and commander creation, stakeholder notification, mandatory review.
Decision
Severity is a declared field on the incident, initialised from the highest proposed severity among its alerts and changed only by an attributed event with a reason. Each level maps to an obligations profile: acknowledgement target and escalation step length, channel policy, whether a Mattermost channel and incident commander role are created, the stakeholder update cadence and whether a review is mandatory. Changing severity re-evaluates the profile immediately.
How it works on-premise
Obligation profiles are versioned control-plane objects published to the paging domain alongside escalation policies, so the dispatcher applies SEV1 channel policy even when the control plane is down. The incident service applies the non-paging obligations (channel, roles, stakeholder cadence, review) when it reconciles the severity event. Email-sourced alerts cannot propose SEV1.
Options weighed
  • ChosenDeclared, attributed, binding obligations: Humans correct the rule's guess, and everything that should follow from severity does so automatically.
  • RejectedComputed from alert signals only: Consistent and hands-free. Wrong precisely when the incident is novel, which is when severity matters most.
  • RejectedSeverity as a label with no obligations attached: Flexible. Every SEV1 depends on someone remembering to open a channel, name a commander and start updates.
Consequences
What it buys
  • A responder raising severity at 03:15 gets a channel, a commander role and a stakeholder clock without asking
  • Every severity change and its reason is in the timeline for the review
  • SEV1 paging behaviour does not depend on the control plane
What it costs
  • Severity inflation by cautious responders produces more channels and reviews
  • Obligation profiles are policy that someone must own and revisit
  • Downgrading severity mid-incident stops obligations that people may have been relying on, so the change is announced in the channel
Choose differently when
In a highly automated environment where severity maps directly to measured customer impact, such as error-budget burn per service, computed severity with a human override may be more consistent.
Why it holds up over time
Four levels and a profile per level are an organisational contract that outlasts tools. The profiles are data, so changing what SEV2 means next year is an edit, not a release.
LessonLet signals propose and people decide, and make the decision do the paperwork.
Shown on views05 19 29

Schedules and escalationWhere on-call is resolved, how time zones are made boring, what escalation advances on, how timers survive a crash, and what a handoff does to a live page.

ADR-14

On-call is resolved ahead of time into 15-minute coverage buckets, 72 hours out

Accepted

Where is the on-call schedule resolved: at page time, or ahead of time?

Context
Page-time resolution is always current but puts rotation layering, overrides, timezone rules and the schedule database inside the paging path. A pre-materialised snapshot makes paging a single read against a replicated structure, at the cost of a staleness window and a second thing that can be wrong. The requirement asks for at least 60 minutes of materialised coverage, O(1) lookup regardless of layering, and staleness under 60 seconds after a change.
Decision
Resolve schedules in the control plane into coverage buckets keyed by rotation and 15-minute UTC interval, each holding the ordered layers and the verified contact methods for that interval. Materialise 72 hours ahead, re-resolve only the affected rotation on any change, verify with an independent resolver (ADR-15), and publish with a version into the paging domain. The dispatcher reads one bucket. If buckets run out, the last available bucket keeps serving and the platform rotation is paged about staleness.
How it works on-premise
The schedule resolver is a Go service reading the schedule store in PostgreSQL. Buckets are JetStream key-value entries named coverage.<rotation>.<bucket>, written with quorum, around 7,000 live entries across 25 rotations. Escalation policies and obligation profiles are published the same way. Change events from the schedule editor trigger resolution within a second; a full re-resolution runs every 15 minutes as a safety net.
Options weighed
  • ChosenAhead of time, 72 hours, verified, one read: Paging reads a small replicated structure and survives three days of control-plane outage with correct coverage.
  • RejectedResolve at page time from the schedule database: Always current and simplest to reason about. Places the database, the layering logic and timezone rules inside every page.
  • RejectedAhead of time, 60 minutes: Meets the letter of the requirement with the least data. Turns a one-hour control-plane outage into stale coverage and leaves no time to verify a DST boundary.
  • RejectedResolve at page time from a local replica of the schedule store: Removes the network dependency. Keeps all the resolution logic, and its bugs, in the paging path.
Consequences
What it buys
  • A deeply layered rotation pages as fast as a flat one
  • Coverage for any past page is reproducible from the snapshot version stamped on it
  • DST boundaries and gaps are resolved and checked days before they matter
What it costs
  • A change can take up to 60 seconds to reach paging, so the outgoing responder may get one more page
  • Two representations of coverage to keep consistent and monitor
  • Contact data is replicated into the paging path, encrypted (ADR-28)
Choose differently when
A handful of flat rotations in one timezone, with a schedule store that is itself outside the estate's failure domain, does not need materialisation; the complexity pays for layering, timezones and independence.
Why it holds up over time
The bucket is a contract: rotation, interval, ordered targets, version. Scheduling rules can grow new layer types, fairness algorithms or calendar integrations, and none of that reaches the paging path, which will read the same bucket in ten years.
LessonCompute the hard answer early, check it, and hand the critical path a lookup.
Shown on views02 06 17
ADR-15

Timezones are resolved in IANA zones, checked by a second resolver, and gaps are found 14 days out

Accepted

How does the platform avoid an uncovered hour or a double page at a daylight-saving transition?

Context
Twenty-five rotations across three timezones hand over in local time, and twice a year some handoff times do not exist or happen twice. Timezone rules change with little notice when governments decide. Schedule bugs are silent until the hour they affect, which is usually a night. The requirement asks for DST-correct resolution tested against boundaries, gap detection 14 days ahead, and snapshots verified before the boundary.
Decision
Define rotations in local wall-clock time in a named IANA zone and store every materialised instant in UTC. Specify the rules for non-existent local times (next valid instant) and repeated ones (first occurrence). Implement resolution twice, independently, and publish buckets only when both agree; on disagreement, hold the previous verified buckets and notify the rotation owner. Run gap detection hourly over 14 days. Pin tzdata per release and roll updates out as releases verified the same way.
How it works on-premise
The primary resolver is Go using its embedded time zone database. The verifier is a separate implementation in Python using zoneinfo, written by a different engineer from the written rules rather than from the Go code, and run as its own service. The release pipeline resolves every rotation across every transition in its zone to 2030 with both. Gap warnings go to the rotation owner by email and chat; gaps inside 24 hours also open a SEV4 against the rotation.
Options weighed
  • ChosenIANA zones, written rules, two implementations, 14-day gaps: Disagreement between independent implementations catches the class of bug that tests written by the same author do not.
  • RejectedOne resolver with extensive DST tests: Half the code. Tests encode the author's understanding of DST, which is the thing most likely to be wrong.
  • RejectedRotations defined in UTC: No DST logic at all. Handoffs drift by an hour twice a year in people's lives, and people will override them by hand, reintroducing the problem.
  • RejectedCalendar-based schedules via iCalendar feeds: Familiar to users. Recurrence and zone handling vary by producer, and the platform would be verifying someone else's resolver.
Consequences
What it buys
  • A handoff at a time that does not exist is found days early, with both answers shown
  • Zone rule changes from upstream become an ordinary release
  • Coverage gaps reach the person who can fix them before they reach a page
What it costs
  • Two implementations of the most intricate logic in the control plane
  • A disagreement freezes publication for that rotation until someone looks
  • A Python service in an otherwise Go control plane
Choose differently when
Rotations all in zones without DST, with handoffs at fixed UTC-friendly hours, get almost nothing from the second resolver and can drop it.
Why it holds up over time
Time rules will keep changing because they are political. Written resolution rules and two independent implementations turn every future change into a disagreement that is visible, instead of a page that silently goes to nobody.
LessonFor logic that is easy to get subtly wrong and rarely exercised, two independent implementations that must agree are worth more than one well-tested one.
Shown on views06 17 24
ADR-16

Escalation advances only on absence of acknowledgement; delivery failure moves within a step

Accepted

What does escalation advance on: absence of acknowledgement, or delivery failure?

Context
Non-acknowledgement is the honest signal but waits the whole step interval even when the platform knows delivery failed. Advancing on delivery failure is faster and wrong whenever a receipt is wrong, and receipt fidelity varies greatly: push acceptance by Apple says nothing about the phone, some SMS carriers report delivered on hand-off, voice ring and answer are reliable but a voicemail answer is not a human. A premature escalation wakes a second person and teaches the first that their pager is optional.
Decision
Use two mechanisms with different powers. The step timer, and only the step timer, advances to the next escalation step, and only if no acknowledgement has been recorded. Within a step, delivery evidence drives a fast path: a hard failure retries the same channel on the other route; silence past the channel's threshold moves to the next channel in the step's channel policy; a target with no deliverable method at all falls immediately to the next target. A coverage gap, no resolved responder at all, falls immediately to the policy's final step.
How it works on-premise
The dispatcher owns the within-step fast path using per-carrier thresholds from continuously measured route health (ADR-21). Escalation timers own step advancement (ADR-17). Voice acknowledgement requires a DTMF keypress after the summary is read, so a voicemail pickup is recorded as answered-not-acknowledged. Receipts and outcomes are written to the ledger and never read by the timer.
Options weighed
  • ChosenTimer advances steps; delivery evidence moves within a step: Fast reaction to known failures without ever trusting a receipt to decide that a human did or did not see a page.
  • RejectedOne timer, no fast path: Simplest and honest. Waits five minutes on an SMS the carrier refused in the first second.
  • RejectedAdvance to the next step on confirmed delivery failure: Fastest escalation. Wakes a second person whenever a receipt is wrong, which on some routes is often.
  • RejectedStop escalation on confirmed delivery: Saves noise. A delivered page to a sleeping person is the ordinary case this platform exists to handle.
Consequences
What it buys
  • Known delivery failures are acted on within seconds, inside the same step
  • Nobody is escalated past because a carrier misreported
  • Step timing is predictable for responders and managers
What it costs
  • A responder whose every channel is silently failing is not escalated past until the step ends
  • Per-carrier thresholds need continuous measurement
  • Voicemail pickups cost a full step of waiting unless another channel reaches the person
Choose differently when
Where the platform owns an end-to-end delivery receipt it trusts, such as an app-level read receipt from its own client, a step could safely advance early on a confirmed failure to that client specifically.
Why it holds up over time
'Acknowledgement is the only proof' is a principle that survives every change of channel. New channels, a smartwatch, a pager network, a messaging app, slot into the within-step fast path with their own thresholds without touching what advances a step.
LessonLet evidence speed up what you do next, but only let proof decide that you are done.
Shown on views19 20 25
ADR-17

Escalation timers are sharded in-memory wheels over durable timer records, taken over by lease

Accepted

Where do escalation timers live, and how do they fire on time after the process holding them dies?

Context
Peak load is modest, 120 open incidents with several targets and channels each, but a single timer that does not fire is a page that does not escalate. Accuracy must be ± 5 s at p99, a timer must survive a process restart and a host loss, cancellation by acknowledgement must take effect within 5 s, and a duplicate fire after failover is acceptable under the duplicate-over-miss principle. The candidates are an in-memory wheel with a write-ahead log, scheduled messages on a durable queue, or rows polled by a scanner.
Decision
Store every timer as a durable record in the paging domain before it is armed. Partition timers into 64 shards by incident id. Each shard is owned by one timer process through a lease; the owner loads the shard's records into a one-second timer wheel and fires from memory. Before firing it re-reads the incident state and skips cancelled or acknowledged work. When an owner stops renewing, another process takes the lease and rebuilds the wheel from the records. Fired timers are marked done with a compare-and-set; a timer that was fired but not marked may fire again.
How it works on-premise
Timer records and shard leases are JetStream key-value entries in the paging domain. Leases are renewed every second and expire after three seconds. Every paging host runs a timer process and competes for shards, so shards spread across all three cells; the process prefers shards whose incidents belong to its own cell's grouping claims. Rebuilding a shard from records takes well under a second at the expected volume. In minority mode, a cell's timer processes keep firing the shards they hold and adopt orphaned shards from their local copy.
Options weighed
  • ChosenSharded wheels over durable records, lease takeover: Accurate to a second from memory, durable in the replicated store, recoverable in seconds, horizontally scalable by shard.
  • RejectedScheduled messages on a durable queue: Lets the broker own time. Cancellation of a scheduled message is awkward or absent in most brokers, and JetStream's scheduling support is too new to carry the ± 5 s promise.
  • RejectedDatabase rows polled by a scanner: Simple and durable. Accuracy is the poll interval plus scan time, and the scanner's database becomes a paging-path dependency.
  • RejectedIn-memory timers without durability: Trivially accurate. A restart during an escalation loses the escalation.
Consequences
What it buys
  • Timers fire within about a second of due in normal operation, and within the lease expiry plus rebuild after a failure
  • Adding hosts spreads shards without any repartitioning of data
  • Cancellation is a state check at fire time, so an acknowledgement stops work already queued
What it costs
  • A timer may fire twice around a takeover; the human-side dedup absorbs most of those
  • Lease and shard logic is custom code on the most critical path, tested by 1,000-takeover runs in the proof phase
  • Clock discipline on paging hosts is part of timer correctness
Choose differently when
If a future broker offered scheduled delivery with cancellation, replication and published accuracy, handing time to the broker would remove the most intricate custom code in the paging path.
Why it holds up over time
The durable record is the source of truth and the wheel is a cache. Either can be replaced independently: a different store for records, a different scheduling structure in memory. The contract, a timer exists before it is armed and state is checked before it fires, carries over.
LessonKeep time in memory for accuracy and on disk for truth, and design so that firing twice is harmless.
Shown on views09 19 34
ADR-18

A handoff adds the incoming responder to a live page; it never removes the outgoing one

Accepted

What happens to an unacknowledged page when a shift ends in the middle of its escalation?

Context
Targets are resolved when a step fires. If the shift changes before acknowledgement, re-resolving would move the page to the incoming responder, who has no context and may not be settled in, and silence the outgoing responder, who may be about to acknowledge. The requirement forbids transferring a triggered page to the incoming responder without continuing to page the outgoing one.
Decision
When a step's targets change because coverage changed during an unacknowledged escalation, the escalation's target set becomes the union of the targets at trigger time and the current targets. Both are paged until anyone acknowledges. The handoff is recorded as an event on the incident. Acknowledged incidents are not re-targeted at handoff; ownership of an acknowledged incident moves only by explicit reassignment.
How it works on-premise
The escalation state in the paging domain stores the snapshot version and targets used at trigger. At each fire, the dispatcher reads the current bucket, computes the union, and records added targets as events. Human-side dedup keys include the responder, so the union never pages the same person twice on one channel within 60 seconds.
Options weighed
  • ChosenUnion of old and new targets until acknowledged: No gap at the boundary and nobody silently dropped, at the cost of one extra person being woken.
  • RejectedPin targets at trigger time: Stable and simple. An escalation that started at 07:59 keeps paging someone who went off shift an hour ago.
  • RejectedRe-resolve to the incoming responder only: Clean ownership. Silences the person most likely to acknowledge, at the worst moment.
Consequences
What it buys
  • Shift boundaries cannot create an unpaged minute
  • The timeline shows exactly who was added at the handoff and why
  • Outgoing responders who are already working the page are not cut off
What it costs
  • Two people are paged for incidents that span a handoff
  • Rotations with frequent handoffs see slightly more interrupts in reporting
  • Explicit reassignment is required for acknowledged incidents, which is one more action for the outgoing responder
Choose differently when
A rotation with a formal, attended handover meeting where live incidents are transferred verbally may prefer pinning to the outgoing responder and an explicit transfer step.
Why it holds up over time
The rule is about people at boundaries, not about any schedule model. New layer types, follow-the-sun rotations or automated shift swaps all meet the same rule at the moment coverage changes under a live page.
LessonAt a boundary, add rather than replace. Removing someone from a live problem should always be a deliberate act.
Shown on views06 19

NotificationWhether to own delivery or aggregate it, how providers fail over, what counts as a duplicate, how every channel acknowledges, and what happens when nobody does.

ADR-19

Deduplicate at the human, not at the sender; prefer sending twice to not sending

Accepted

How does an at-least-once paging path avoid spamming a responder without risking a missed page?

Context
Every layer of the paging path retries: the gateway, the engine, the timers after takeover, the dispatcher across routes, and minority cells during partitions. Exactly-once delivery to a phone is not achievable across carriers, and trying for it at the sender means sometimes not sending. The requirement asks that a responder never receive the same page twice on the same channel within 60 seconds, and that the platform prefer sending twice over risking not sending.
Decision
Allow every component to retry freely. Immediately before a send, the dispatcher claims a key made of responder, channel, incident and step with a 60-second expiry. If the claim fails because the key exists, the send is skipped and recorded as a deduplicated attempt. If the claim cannot be made at all, because the paging domain is unavailable or the cell is in minority mode, the dispatcher sends anyway and records that deduplication was unavailable.
How it works on-premise
Human-dedup keys are JetStream key-value entries with a 60-second per-key TTL in the paging domain, created with create-if-absent semantics. In minority mode the dispatcher uses a local in-memory dedup table, which prevents duplicates within the cell but not across the partition. Every skipped and every unguarded send is a ledger entry with its reason.
Options weighed
  • ChosenDedup at the human with a 60-second claim; send when unsure: Retries stay cheap and safe everywhere upstream, and the one place that touches a person enforces the rule.
  • RejectedExactly-once dispatch with a transactional outbox: Clean semantics inside the platform. The carrier boundary is still at-least-once, and a stuck transaction means no page.
  • RejectedDedup at each component: Stops duplicates early. Spreads the rule across five services, each with its own idea of what a duplicate is.
Consequences
What it buys
  • Takeovers, failovers and replays do not need careful exactly-once handling
  • The 60-second rule is enforced in one place and is testable there
  • Every duplicate that does reach a person is explained in the ledger
What it costs
  • Partitions and paging-domain outages can produce duplicates within 60 seconds
  • Dedup keys depend on a stable definition of step, which constrains policy editing mid-incident
  • A same-channel retry on another route after a hard failure must not be deduplicated, so the claim is released on confirmed failure
Choose differently when
If duplicate notifications carried a real cost, such as per-message charges large enough to matter at volume or a regulatory limit on repeated contact, stronger sender-side controls would be justified.
Why it holds up over time
The principle and the key definition are independent of channel and carrier. New channels get a key and a window; the rule that uncertainty resolves toward sending does not change.
LessonPut the idempotency check next to the side effect that matters, and decide in advance which way it fails.
Shown on views09 12 20
ADR-20

Aggregate carriers, own the last software hop: Asterisk, Jasmin, a push relay and a responder app

Accepted

Is the platform a notification provider, or an aggregator of providers?

Context
Owning delivery end to end is impossible on-premises: phones are reached through carriers, and iOS and Android devices are woken only through Apple and Google. Aggregating third-party notification services is cheap and makes provider diversity easy, but receipts are inconsistent, acknowledgement arrives through someone else's webhook, and do-not-disturb bypass is out of the platform's hands. The requirement needs acknowledgement from every channel paged on, SEV1 DND bypass, and sub-15-second p95 delivery.
Decision
Contract two carriers directly for SIP trunking and SMPP, and run the software that speaks to them: Asterisk for voice with an IVR that reads the summary and takes a keypress, Jasmin for SMS including inbound replies. Run a push relay that talks to APNs and FCM directly. Ship a thin responder app of the platform's own that receives critical alerts, shows the summary and acknowledges with one button. Use Postfix through two smarthosts for email.
How it works on-premise
Asterisk with PJSIP trunks to carrier A from cells A and B and to carrier B from cell C, using ARI to hand DTMF to the ack receiver. Summaries are rendered to audio ahead of time with eSpeak NG when the service catalogue changes, not at call time. Jasmin binds SMPP to both carriers with MO messages routed to the ack receiver. The push relay uses APNs token authentication and FCM HTTP v1 with credentials from OpenBao. The app requests Apple's Critical Alerts entitlement and Android's notification policy access for SEV1.
Options weighed
  • ChosenDirect carrier contracts, owned voice, SMS and push software, own app: The platform receives every acknowledgement itself, controls receipts and timing, and can pursue DND bypass for its own app.
  • RejectedAggregate cloud notification APIs: Far less to build and trivially diverse. Puts a cloud service on the paging path, which the constraint forbids, and moves acknowledgement fidelity out of reach.
  • RejectedSelf-hosted push via UnifiedPush or a relay like ntfy: Keeps push inside the organisation. iOS still requires APNs to wake the device, and a third-party app cannot hold the critical-alert entitlement on the platform's behalf.
  • Right elsewhereGSM modem banks as the primary SMS route: Fully owned hardware. Throughput and carrier policy toward modem traffic make it a fallback, which is where it sits (ADR-30).
Consequences
What it buys
  • A keypress, an SMS reply and an app button are all acknowledgements the platform receives on its own infrastructure
  • Receipt timing and failure codes are measured per route, not reported by a middleman
  • Carrier diversity is a contract and a trunk, not a vendor's roadmap
What it costs
  • Carrier contracts, number provisioning and, in some jurisdictions, sender and template registration are the platform team's work
  • An app to maintain on two mobile platforms
  • If Apple refuses the entitlement, iOS SEV1 bypass relies on voice from a number responders set as an emergency bypass contact
Choose differently when
An organisation permitted to use cloud services, with responders in regions where a reputable notification API meets the latency target, should aggregate and spend the saved effort on alert quality.
Why it holds up over time
SIP, SMPP, SMTP, APNs and FCM change slowly and are replaceable one route at a time. The acknowledgement contract, a signed token that any channel can return to the ack receiver, is independent of which of them delivered it.
LessonYou cannot own the network to the phone. You can own everything up to it, and every acknowledgement coming back.
Shown on views04 08 19
ADR-21

Fail over on the same channel after a hard failure, to the next channel after silence

Accepted

How does the dispatcher use two independent providers within one escalation step without waiting for the step interval or duplicating pages?

Context
The requirement asks for failover between providers within a step when the primary reports failure or exceeds a latency threshold, and for provider health measured continuously rather than at page time. It also forbids the same page twice on the same channel within 60 seconds. A latency-triggered failover to the other provider on the same channel will produce exactly that duplicate whenever the first provider was merely slow.
Decision
Treat the two triggers differently. On a hard failure, a NACK, throttle, refused bind or refused call, retry the same channel on the other route immediately; the first attempt demonstrably did not leave. On silence past the channel's threshold, move to the next channel in the step's policy on the healthier route, not to the same channel on the other route. Route health, measured by loopback traffic, decides which route is tried first.
How it works on-premise
Loopback traffic from each cell: an SMS every ten minutes per route to a platform-owned SIM in the GSM bank at site C, and a call every fifteen minutes per route to a DID answered by Asterisk at another cell. Latency, receipt rate and DTMF success feed a route score in the paging domain. Thresholds start at 20 s for SMS receipts, 10 s to ringing for voice and 3 s for push acceptance, and are tuned from the proof phase measurements.
Options weighed
  • ChosenSame channel on failure, next channel on silence: Satisfies both the failover requirement and the 60-second rule, and escalates the medium rather than repeating it.
  • RejectedSame channel on the other route on any trigger: The literal reading of the requirement. Sends a duplicate SMS every time a route is slow rather than failed.
  • RejectedSend every channel on both routes at once for SEV1: Maximum reach. Violates the duplicate rule on every page and doubles the most expensive traffic.
  • RejectedHealth checked at page time: Always current. Adds seconds to the first attempt and says nothing about the route's recent behaviour.
Consequences
What it buys
  • A failed carrier costs a page seconds, not a step
  • Slow routes produce a call rather than a repeated SMS, which is also more likely to wake someone
  • Route health is known before a real page needs it
What it costs
  • Synthetic loopback traffic is a standing cost on both carriers
  • Silence thresholds that are too tight produce more calls than needed
  • Channel policy must order channels so that 'next' is always meaningful
Choose differently when
Where every route returns reliable, fast delivery receipts, a latency failover on the same channel becomes safe and simpler.
Why it holds up over time
The distinction between 'did not leave' and 'has not arrived' is true of every messaging system that will ever exist, and so is the answer: repeat what failed, change what is slow.
LessonTreat 'it failed' and 'it has not worked yet' as different events. They need different responses.
Shown on views20 26 34
ADR-22

Every channel acknowledges with a signed, single-use token; no login, no DNS dependency

Accepted

How does a responder stop an escalation from any channel they were paged on, with no web session and possibly no working corporate infrastructure?

Context
The requirement asks that acknowledgement work from every channel, a single keypress on a call and a reply on SMS, and not require a web session. Responders acknowledge from bed, from a car and during the identity provider's outage. An acknowledgement mechanism that needs login, VPN, corporate DNS or the console fails in exactly the scenarios that produce pages.
Decision
Every notification carries an acknowledgement token: a signed structure naming incident, responder, step and expiry, verifiable by any cell. Voice acknowledges by DTMF after the summary, SMS by replying with a short code tied to the token, push by an app button that posts the token, and email by a link or a reply. The ack receiver verifies the token, performs the state compare-and-set and marks the token used. The responder app carries the addresses of all three cells and tries them in order, so neither DNS nor a single cell is required.
How it works on-premise
Tokens are signed with Ed25519 keys held by every cell, rotated monthly with overlap. SMS short codes map to tokens in the paging domain for the lifetime of the step. Ack endpoints are exposed on each cell's public address behind HAProxy with rate limiting per source. The app pins the platform's CA and the three cell addresses in its build, updated by app release. Acknowledgements in minority mode are recorded locally and replayed.
Options weighed
  • ChosenSigned single-use tokens on every channel, three hard-coded endpoints: Acknowledgement depends only on reaching one cell, and a leaked token can do one thing once.
  • RejectedAcknowledge through the console after login: Strong authentication. Fails during identity outages and costs a groggy responder a minute of typing.
  • RejectedUnsigned reply matching by phone number: Simplest for SMS. Caller ID and sender numbers can be spoofed, and one number may be on several rotations.
  • RejectedChat-based acknowledgement only: Convenient where chat is always open. Chat is a control-plane integration and may be the thing that is down.
Consequences
What it buys
  • Acknowledgement works throughout control-plane, identity and DNS outages
  • A forwarded or intercepted token can acknowledge one step of one incident and nothing else
  • Every acknowledgement records its channel, cell and token, for the review
What it costs
  • Cell addresses changing requires an app release, so they are chosen to be stable
  • SMS short codes are guessable if too short; they are six characters and rate limited
  • Token signing keys are one more secret to rotate across three cells
Choose differently when
A platform whose responders always work from managed devices on a network that is itself outside the estate's failure domain could safely rely on the console and single sign-on for acknowledgement.
Why it holds up over time
The token format is channel-independent. A new channel only needs a way to carry six characters or a URL back to a cell; the verification and state change do not move.
LessonMake the action that stops the alarm need as little of the world as possible, and make it useless for anything else.
Shown on views04 19 32
ADR-23

Escalation exhaustion is a terminal state, a repeating broadcast and an incident against the platform

Accepted

What happens when every step of an escalation policy has run and nobody has acknowledged?

Context
The requirement calls this the single worst outcome the platform can produce and asks for an explicit terminal state treated as an incident against the platform. Many tools close or leave such incidents silently, which means the organisation discovers the gap the next morning. Exhaustion is almost always a configuration failure, a wrong rota, stale contact data or a policy that ends too early, rather than bad luck.
Decision
Require every escalation policy to end in a final step; the editor refuses to save one without it, and the default final step is the owning team's management layer plus a broadcast to every responder in the owning team. Keep paging earlier targets as later steps begin. On exhaustion, set the escalation state to exhausted, repeat the broadcast every 5 minutes until acknowledged, and open a SEV2 incident against the platform on carrier B to the platform rotation.
How it works on-premise
Exhausted is a state in the paging domain, not a flag, so timers and dispatchers treat it as active. The platform incident is opened by the timer service directly through the engine's API in the same cell, and its dispatch is pinned to carrier B routes. The review template for exhaustion incidents includes the rotation, the contact methods tried and their ledger outcomes.
Options weighed
  • ChosenTerminal state, repeating broadcast, platform incident: The worst outcome becomes the loudest one and lands on the team that can fix its cause.
  • RejectedLoop the policy from step one: Keeps trying the same people. Does not reach anyone new and hides that the policy failed.
  • RejectedLeave the incident open and stop paging: Avoids noise. Produces exactly the silent failure the requirement names.
Consequences
What it buys
  • No incident can end in silence
  • Exhaustion produces a review of the rota and contact data, which prevents the next one
  • Management learns about exhaustion from the platform, not from a customer
What it costs
  • A whole team can be woken by one person's dead phone
  • Repeat broadcasts are expensive in voice minutes during a long gap
  • Exhaustion incidents count against the platform's own reliability reporting
Choose differently when
None in this organisation. A platform that pages humans has no configuration in which silent exhaustion is the better outcome.
Why it holds up over time
The rule is independent of how policies are modelled or who is on them. Future policy features, such as follow-the-sun or automated backfill, must still end somewhere loud.
LessonName your worst outcome, make it a state rather than an absence, and route it to the people who can prevent it.
Shown on views25 34

Response and learningHow the response is coordinated around the incident record, and how a review is made the cheapest thing to finish.

ADR-24

Coordination happens around the record: channels, roles, updates and ownership as events

Accepted

How are channels, roles, stakeholder updates and ownership handled so that coordination helps the response and the record at the same time?

Context
Above a severity threshold the requirement asks for a durable channel linked to the incident, roles held by one person at a time with handovers recorded, paging people in by name, role or service ownership, stakeholder updates as first-class objects with an audience and a next-update-by time, and routing derived from a service catalogue. Chat is also the tool most likely to be unavailable or overwhelmed during a large outage.
Decision
Create a Mattermost channel per qualifying incident asynchronously from the incident service, record the channel id on the incident, and retry until it exists; channel creation never blocks paging. Model roles, handovers, pages-in and stakeholder updates as incident events through the API. Stakeholder updates use templates, a declared audience and a next-update-by time that raises a reminder to the communications lead. Derive ownership, escalation policy and runbook from a service catalogue kept as YAML in Git and synchronised into the control plane.
How it works on-premise
Mattermost Team Edition on the organisation's existing deployment, reached through a chat bridge service with a bot account. Chat messages are not state; a slash command that changes state calls the API as the invoking user. Catalogue files follow the Backstage catalog-info format so the organisation's developer portal can read the same source. Stakeholder updates are emailed to their audience and published to the static status page at site C when marked external.
Options weighed
  • ChosenEvents through the API, chat as an asynchronous integration, catalogue in Git: The record captures coordination without depending on chat, and ownership has one reviewed source.
  • RejectedChat as the system of record for the response: Where responders already are. Unstructured, unattributed state and a hard dependency on the chat product during the incident.
  • RejectedOwnership entered per alert rule: Local and flexible. Copies drift from reality within months, which is the failure the requirement names.
  • DeferredAdopt Backstage as the catalogue runtime: A full developer portal. The platform needs the data, not the portal; the shared file format keeps the option open.
Consequences
What it buys
  • Who held which role when, and what stakeholders were told when, are in the timeline
  • A chat outage delays a channel, not a page or a record
  • Ownership changes are reviewed changes to one file
What it costs
  • Responders must use commands or the console for state changes rather than free text
  • Catalogue quality becomes a dependency for routing, and needs an owner
  • Stakeholder templates need maintaining as the organisation's audiences change
Choose differently when
A small organisation where everyone is in one channel and there are no external stakeholders can let chat carry coordination and record only pages and state in the platform.
Why it holds up over time
Chat products change every few years; events about roles and updates do not. The bridge is the only component that knows which chat product exists, and the catalogue's file format is shared with tools beyond this platform.
LessonLet people coordinate wherever they talk, but record what they decided somewhere that does not depend on where they talk.
Shown on views05 10 11
ADR-25

Reviews are generated from the log, blameless by template, with action items tracked both ways

Accepted

How does the platform make the post-incident review the cheapest path, and make it change something?

Context
Reviews that start from a blank page happen late or not at all. Reviews that name people teach everyone to hide near misses. Action items filed into a tracker the platform cannot see die quietly, or are closed without any change. The requirement asks for an automatic draft with computed intervals, a template with no individual attribution field, action items synchronised with the issue tracker, overdue reviews escalated, and search across reviews for recurring factors.
Decision
Generate a draft on resolution for every incident at or above the mandatory threshold, with timeline, responders, linked changes and the intervals to detect, acknowledge, mitigate and resolve computed from events. Structure the template around contributing factors and system conditions from a controlled vocabulary plus free text, with no field for a person. Model action items as objects synchronised both ways with the tracker, and report items overdue or closed without a linked change. Index reviews for full-text and factor search in PostgreSQL.
How it works on-premise
The review module is a control-plane service writing review edits as events. Tracker synchronisation uses the tracker's webhooks inbound and REST API outbound, with a nightly reconciliation job that repairs drift and reports conflicts. Search uses PostgreSQL full-text search with pg_trgm for fuzzy matching, which comfortably covers a few thousand reviews a year. Overdue SEV1 and SEV2 reviews notify the owning team's management layer by email and chat, never by page.
Options weighed
  • ChosenGenerated draft, blameless template, two-way action items, PostgreSQL search: The review starts from facts, cannot name a person, and its actions stay visible until they change something.
  • RejectedReviews as documents in a wiki: Flexible and familiar. No computed intervals, no action tracking, and search across reviews is a hope.
  • RejectedOne-way export of action items to the tracker: Simpler. The platform loses sight of whether anything was done.
  • RejectedOpenSearch for review search: Better relevance at scale. Unnecessary for thousands of documents, and one more cluster to run.
Consequences
What it buys
  • The first draft costs the owner minutes, not hours
  • Recurring contributing factors become countable across incidents
  • Action items closed with no change are reported as such
What it costs
  • The controlled vocabulary needs curation or it becomes a junk drawer
  • Two-way sync with a tracker is a long-lived integration with conflict handling
  • Some teams will want a free attribution field and will need the reason explained
Choose differently when
An organisation with a mature review practice in an existing tool, with action tracking that already works, should integrate the computed timeline into that tool rather than move reviews.
Why it holds up over time
The event log keeps the review's facts reproducible whatever the review tool becomes, and blamelessness is structural, so it survives changes of team, leadership and fashion.
LessonMake the right review the easiest one to write, and make the wrong one impossible to write in the template.
Shown on views07 15 31

Security and identitySigning in when the identity provider is down, integrations that are never trusted, contact data that can be erased from an immutable system, and incidents only some people may read.

ADR-26

Keycloak brokers the corporate identity provider and holds break-glass accounts on hardware keys

Accepted

How do responders sign in to the console when the organisation's identity provider is itself the outage?

Context
Federated authentication is required, and the corporate identity provider will sometimes be down, sometimes during the incident it is causing. The requirement asks for a break-glass path for the on-call console usable in that case, authorisation scoped by team and service, and every sensitive action audited.
Decision
Put Keycloak between the console and the corporate identity provider. Normal logins federate. A small set of local break-glass accounts, each bound to registered WebAuthn hardware keys, can sign in when federation fails. Break-glass sessions last four hours and carry a responder role that can acknowledge, resolve, change severity and post updates, but not edit schedules, escalation policies or suppression rules. Every break-glass login is an audit event and a notification to the security rotation.
How it works on-premise
Keycloak in the control-plane cluster with its database on the control-plane PostgreSQL. Groups from the identity provider map to team and service scopes through the service catalogue. Break-glass accounts, about a dozen across three timezones with two keys each, are reviewed quarterly and exercised in the control-plane rebuild drill. Keycloak is not used anywhere on the paging path.
Options weighed
  • ChosenKeycloak broker with WebAuthn break-glass accounts: One place that knows both login paths, with local credentials that are phishing-resistant and few.
  • RejectedConsole federated directly to the identity provider: One less component. No login at all during an identity outage.
  • RejectedShared emergency password in a safe: Simple. Unattributable, hard to rotate and unavailable to whoever is not near the safe at 03:00.
  • RejectedLocal accounts for every responder: Always available. Four hundred parallel credentials to provision, offboard and audit.
Consequences
What it buys
  • An identity outage costs convenience, not the ability to run a response
  • Break-glass cannot be used to make the damaging configuration changes
  • Every use is attributable to a person and a hardware key
What it costs
  • Keycloak is one more service to patch and operate
  • Hardware keys must be issued, registered and replaced
  • Break-glass holders are a small group whose availability must be planned
Choose differently when
If the identity provider were demonstrably independent of the estate's failure domain, a cloud identity service used only for this platform, for example, the break-glass path could shrink to a documented emergency procedure.
Why it holds up over time
OIDC and WebAuthn are open standards with long horizons. The identity provider behind Keycloak can change without the console changing, and the break-glass rule, fewer powers and more scrutiny, is independent of the broker.
LessonEvery login path that depends on the thing that might be broken needs a narrower one that does not.
Shown on views22 33
ADR-27

Every integration is untrusted, independently authenticated and revocable without a deploy

Accepted

How are inbound integrations authenticated so that one compromised or broken integration affects only itself?

Context
Forty services and their monitoring, vendors and ticketing systems will integrate with the platform, owned by many teams with different security practices. A shared ingest token turns any leak into the ability to page anyone. The requirement asks for per-integration credentials, signature verification, independent revocation, rotation without deployment, and provider credentials held in a secret store.
Decision
Give every integration its own identity, a signing secret for HMAC over the body and a timestamp, and an allowed set of services it may raise alerts for. Hold up to two valid secrets per integration so rotation overlaps. Store secrets in OpenBao and publish only what the gateway needs to verify into the paging domain. Revocation is a control-plane action that takes effect within the coverage staleness bound. Carrier, APNs, FCM and token-signing credentials are also held in OpenBao and loaded into memory at start.
How it works on-premise
OpenBao key-value secrets for integration secrets and transit keys for wrapping the verification material published to the paging domain. Gateways verify with a 5-minute replay window. Integrations that can present client certificates may use mTLS from a per-integration certificate instead of HMAC. Email integrations authenticate by address and sender allowlist and are limited in the severity they may propose.
Options weighed
  • ChosenPer-integration HMAC or mTLS, two live secrets, OpenBao, scope per service: A leak is contained to one integration's services and ends with a rotation nobody else notices.
  • RejectedOne shared ingest token: Easy to distribute. One leak lets anyone page any rotation, and rotation breaks every integration at once.
  • RejectedNetwork allowlisting only: No secrets to manage. Anything on an allowed network can send anything, and the estate's network is not a trust boundary during an incident.
  • RejectedOAuth client credentials against the identity provider: Standard and centrally managed. Puts the identity provider on the ingest path, which must work while it is down.
Consequences
What it buys
  • A compromised integration can raise alerts only for its own services, and is revoked alone
  • Secret rotation is routine because two secrets overlap
  • Ingest never waits for the identity provider or OpenBao
What it costs
  • Integration owners must implement signing, which some vendors cannot
  • A signing proxy is needed for sources that can only send unsigned webhooks
  • Two secret stores in effect: OpenBao as source, the paging domain as the published copy
Choose differently when
A platform receiving alerts only from a single monitoring stack operated by the platform team itself could authenticate that one source with mTLS and skip per-integration management.
Why it holds up over time
HMAC signatures and mTLS will outlive every monitoring product sending them. The scope-per-integration rule holds however integrations multiply.
LessonGive every sender its own key and its own blast radius, and make revoking one as dull as rotating it.
Shown on views11 16 32
ADR-28

Contact data is encrypted per responder, so offboarding erases it from an immutable system

Accepted

How can personal contact data be restricted, encrypted and erased when the platform keeps an immutable log for seven years and replicates coverage into three sites?

Context
Phone numbers and device tokens are personal data with restricted access and a required erasure path on offboarding. The same data must be in the paging path to be useful, replicated to three sites and cached in coverage buckets. The event log and the ledger are immutable by design, and ledger records name the address a page was sent to.
Decision
Encrypt every contact address with a data key specific to its responder, wrapped by OpenBao's transit engine. Coverage buckets carry method ids and the wrapped key reference, not clear addresses. The event log and ledger store responder ids and method ids only. The dispatcher unwraps and decrypts at the moment of sending. Offboarding deletes the responder's data key, after which every stored copy of their addresses, in any store and any backup, is unreadable.
How it works on-premise
Addresses encrypted with AES-GCM under a per-responder key. Wrapped keys stored beside the contact method in PostgreSQL and replicated into the paging domain. Dispatchers cache unwrapped keys in memory for the duration of a step. Key deletion is an audited control-plane action executed by OpenBao and propagated to caches within the staleness bound. Access to clear contact data in the console is limited to the responder and rotation owners, and every view is an audit event.
Options weighed
  • ChosenPer-responder keys and crypto-shredding: Erasure works across immutable logs, replicas and backups, without mutating any of them.
  • RejectedDelete rows on offboarding: Obvious. Cannot reach immutable logs, replicated buckets or backups, so it is not erasure.
  • RejectedEncrypt with one platform key: Protects data at rest. Erasing one person means re-encrypting or deleting from every store.
  • RejectedKeep contact data only in the corporate directory: One source of truth. Puts the directory on the paging path.
Consequences
What it buys
  • Offboarding erasure is one key deletion, verifiable and auditable
  • Replicas and backups of the paging state leak nothing without the keys
  • The immutable record stays immutable
What it costs
  • Losing OpenBao's transit keys would make every address unreadable, so they are backed up and drilled like the event log
  • Decryption at send time adds a small cost to each dispatch
  • Historical reviews cannot show the actual number used for an offboarded responder
Choose differently when
If contact data were not considered personal data in the jurisdiction, or retention obligations required keeping addresses for the audit period, simpler at-rest encryption with ordinary access control would do.
Why it holds up over time
Crypto-shredding decouples erasure from every storage decision the platform will make later. Any future store that holds contact data inherits erasure for free as long as it holds ciphertext.
LessonWhen a system must never delete and must sometimes forget, encrypt per subject and delete the key.
Shown on views15 32
ADR-29

Incidents are readable by default; sensitive ones are restricted by row-level security, and the restriction is audited

Accepted

How are security and personnel-adjacent incidents kept to a named group without making every other incident harder to reach?

Context
Responders need broad read access to timelines to learn and to help. A small share of incidents, security investigations and anything involving individuals, must be visible only to a named group. Restriction can also be misused to hide an embarrassing incident, so the act of restricting must itself be visible. The requirement asks for readable-by-default timelines, restriction to a named group, and the restriction audited.
Decision
Default every incident to readable by all responders. Allow the incident commander or the security rotation to mark an incident restricted to a named group, as an event with a reason. Enforce restriction in the database with row-level security on the event log and projections, not only in the console. Redact restricted incidents in the outbound stream to id, state and severity. The existence of a restricted incident is visible; its content is not.
How it works on-premise
PostgreSQL row-level security policies keyed on the incident's restriction group and the session's groups, set by the incident service from the Keycloak token. ClickHouse reports exclude restricted incident content. Restriction and unrestriction events are included in a weekly report to the security and reliability leads.
Options weighed
  • ChosenOpen by default, row-level security for restricted, restriction audited: Broad learning for the common case, enforced secrecy for the rare case, and no quiet hiding.
  • RejectedRestricted incidents in a separate tool: Strong isolation. Loses paging, timelines and reviews for exactly the incidents where accuracy matters most.
  • RejectedConsole-level filtering only: Easy to build. Every API, export and report must reimplement it, and one will not.
Consequences
What it buys
  • Restriction holds for every client of the database, including future ones
  • Nobody can make an incident disappear, only restrict who reads it
  • Security incidents use the same paging and review machinery as everything else
What it costs
  • Row-level security adds a policy check to every query on the log
  • Restricted incidents are excluded from cross-incident factor search for most users
  • Groups must be kept accurate in the identity provider
Choose differently when
An organisation whose security incidents are handled by a separate team with its own tooling and legal obligations may be required to keep them out of this platform entirely.
Why it holds up over time
Enforcement in the database survives changes of console, API and reporting tool. The audited restriction event is a governance control that does not depend on any product.
LessonPut access control where every reader has to pass through it, and make the decision to restrict as visible as the thing restricted.
Shown on views15 32

Operations and costHow the platform proves it works from outside itself, how the paging path changes without becoming the outage, and how money and sleep are kept visible.

ADR-30

A synthetic page runs every minute, an outside watchdog pages on carrier B, and a manual fallback kit is drilled

Accepted

How does the platform prove it works, and how is its own failure detected by something that does not share it?

Context
A paging platform that alerts about itself through itself fails silently in the one case that matters. The requirement asks for a continuous end-to-end synthetic page through the real path, its failure paged through the alternate provider, and a manually operated fallback paging path for total primary failure, exercised on a schedule.
Decision
Inject a synthetic alert into every cell every 60 seconds; it becomes a test incident, pages a platform-owned SIM and a test app, and is acknowledged by a robot. Run a dead-man switch at site C that expects heartbeats from every cell and the synthetic loop, and on silence pages the platform rotation through carrier B directly, bypassing the paging services. Keep a manual fallback kit, a hardened laptop with a CLI, an hourly encrypted export of coverage and policies, and the GSM modem bank, and use it once a year to page a volunteer rotation.
How it works on-premise
The synthetic injector and robot run on hosts outside the cells at each site. The dead-man switch is a small independent service at site C with its own Asterisk dialplan and Jasmin route, sharing no process with cell C's paging services. The fallback CLI drives Gammu against the modem bank and reads the exported snapshot decrypted with keys held by two platform leads. Results of every drill are published as incidents with reviews.
Options weighed
  • ChosenMinute synthetic, independent watchdog, drilled manual kit: The platform's most honest SLI, an outside check that cannot share its failure, and a last resort that has been used.
  • RejectedInternal health checks only: Cheap. A process can be healthy while the path through carriers to a phone is broken.
  • RejectedSynthetic pages to real responders: Tests the true last mile. Burns responder trust at one page a minute.
  • RejectedManual fallback documented but not exercised: Costs nothing. Will not work on the day, because the export, the keys or the modems will have drifted.
Consequences
What it buys
  • End-to-end failure is known within about two minutes, from outside the failing system
  • The fallback kit's steps, keys and data are known to work as of the last drill
  • The synthetic time series is a continuous measure of the p95 target
What it costs
  • Standing carrier traffic for synthetics and loopback on both routes
  • Drill time from the platform team and a volunteer rotation each year
  • The watchdog is one more independent component to keep independent
Choose differently when
A smaller platform with a single site could reasonably rely on an external uptime service calling a phone, if a cloud service were permitted for the watchdog alone.
Why it holds up over time
'Test through the real path, watch from outside, exercise the fallback' holds for any future architecture. The synthetic contract, an alert in, an acknowledgement back, is the definition of the product.
LessonThe only proof that a paging system pages is a page. Send one every minute, and have someone else watch for it.
Shown on views28 34
ADR-31

The paging path changes one cell at a time, never during a SEV1 or SEV2, from artefacts already on the host

Accepted

How does the paging path change often enough to stay current without a release becoming the outage?

Context
Changes are the most common cause of outages, and three identical cells upgraded together are one cell. The platform's outage is everyone's outage, and a deploy that depends on the registry, Git or DNS cannot run during the incidents that most need a fix. The paging path still needs regular releases for security patches, tzdata and features.
Decision
Release the paging path through gated waves: staging cells, then cell C, then cell B the next day, then cell A the day after. Each wave requires three synthetic pages acknowledged through the new cell. No wave starts while a SEV1 or SEV2 is open, and none starts unless both platform-team regions are in working hours. Artefacts, configuration and tzdata are staged on every host before the switch, and rollback is a symlink and restart. The control plane deploys continuously with ordinary canaries.
How it works on-premise
Forgejo with signed commits and two reviewers, Forgejo Actions building static binaries, SBOMs and cosign signatures pushed to Harbor. Gates run schedule resolution to 2030 against both resolvers and replay a recorded storm with an injected SEV2. Ansible stages and switches host by host. The freeze check queries the paging domain for open SEV1 and SEV2 incidents directly, so it works when the control plane does not.
Options weighed
  • ChosenGated waves, one cell per day, freeze during major incidents, staged artefacts: A bad release affects one cell, is caught by a synthetic page, and is reversed without any external service.
  • RejectedAll cells at once with fast rollback: Quick. A defect that breaks paging breaks it everywhere, and the rollback needs a working path to run.
  • RejectedBlue-green per cell with traffic shifting: Fine-grained. Doubles hosts per cell for a path with little traffic to shift.
  • RejectedRelease only quarterly: Fewer changes. Larger changes, stale tzdata and delayed security patches.
Consequences
What it buys
  • Most defects are found by a synthetic page in the outpost cell before an estate cell is touched
  • Deploys and rollbacks work during infrastructure incidents
  • Releases are routine enough that nobody fears them
What it costs
  • A full paging-path release takes three working days
  • Urgent security fixes need an expedited path, which still keeps one-cell-at-a-time
  • Freezes during long incidents delay fixes that might help
Choose differently when
If the paging path grew to many cells in many locations, waves by percentage of cells would replace waves by day.
Why it holds up over time
The rules, one failure domain at a time, prove with a real page, never during a major incident, never depend on the network to roll back, apply to any deployment technology the team adopts later.
LessonThe safest release process is the one that assumes the release is wrong and the network is down.
Shown on views27 09
ADR-32

Money and sleep are metered per service and rotation; the independence premium is a stated number

Accepted

How are notification spend, standby cost and human interrupt load kept visible and attributed?

Context
Voice and SMS are the platform's dominant marginal cost and are driven by alert quality, which the platform team does not control. The outpost is insurance whose premium must be a known number. Out-of-hours pages and non-actionable pages are the cost to people, and the requirement asks for them to be measured and reported, not enforced.
Decision
Write a cost on every ledger attempt from the carrier rate card at send time. Attribute spend, pages, out-of-hours interrupts and actionable rate to service, rotation and responder. Deliver one weekly report per service owner combining noise, cost and interrupts. State the outpost's yearly premium in the platform budget as insurance, currently estimated at about USD 55,000 before staff time, and review it yearly against the incidents it covered. Prefer push and email over voice where severity policy allows, and never weaken SEV1.
How it works on-premise
Rate cards are versioned control-plane data. ClickHouse materialised views compute weekly aggregates; a job renders the report and emails it, and Grafana dashboards show the same data. Actionable pages are those followed within the incident by a linked change, a severity change, a note or a recorded decision. Synthetic and loopback traffic are attributed to the platform's own budget.
Options weighed
  • ChosenMetered, attributed, reported weekly; premium stated: Alert hygiene becomes a budget line owned by the service that causes it, and insurance is defended with a number.
  • RejectedCentral platform budget, no attribution: Simple accounting. The teams that generate noise never see its price.
  • RejectedEnforce interrupt limits by suppressing pages: Protects sleep directly. Trades a burned-out rota for a missed incident.
  • DeferredCharge back real money per page: Strong incentive. Risks teams under-alerting to save budget; revisit after a year of reporting.
Consequences
What it buys
  • Service owners see what their alerts cost in money and in their colleagues' nights
  • The outpost survives budget reviews because its cost and value are both visible
  • Severity channel policy can be tuned against measured spend
What it costs
  • Rate cards must be kept current with carrier contracts
  • Actionable is a heuristic and will be argued with
  • Reporting without enforcement depends on management attention
Choose differently when
If carrier traffic were effectively free under a flat contract, spend attribution would matter little and the report could focus entirely on interrupts.
Why it holds up over time
Cost per attempt and interrupts per responder are measures that stay meaningful whatever the carriers, channels or organisation structure become. Stated premiums are how insurance survives changes of leadership.
LessonPut a price on noise where the people making it can see it, and put a price on insurance before anyone asks what it is for.
Shown on views26 29

Every package used, in one table

Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.

PackageWhat it isWhat it does hereConsidered instead
HAProxy TCP and HTTP load balancer Cell edge: TLS termination, per-integration rate limits in stick tables, anycast front for ingest and acknowledgements Envoy, nginx
BIRD Internet routing daemon Announces each cell's anycast address over BGP with BFD, withdrawn when the cell is unhealthy FRRouting
Go Programming language and toolchain Ingest gateway, incident engine, timers, dispatcher, ack receiver and schedule resolver as static binaries Rust, Java
CEL (cel-go) Common Expression Language Sandboxed per-integration alert mappings with cost limits JSONata, Lua
NATS JetStream Messaging system with persistence, CNCF Ingest buffer, paging state, coverage buckets, timers, deduplication, outbox and ledger stream Apache Kafka, etcd
Postfix Mail transfer agent Inbound email alerts and replies; outbound email notifications through two smarthosts Exim
Asterisk Telephony server Voice notifications with pre-rendered summaries and DTMF acknowledgement over SIP trunks FreeSWITCH
eSpeak NG Speech synthesiser Renders service names and summary templates to audio ahead of time Festival
Jasmin SMS gateway SMPP binds to both carriers, delivery receipts and inbound SMS acknowledgements Kannel
Gammu Mobile phone management tool Drives the GSM modem bank for loopback checks and the manual fallback kit Kannel with AT modems
PostgreSQL Relational database Append-only event log, projections, schedule store, reviews and search, with row-level security MySQL
CloudNativePG PostgreSQL operator for Kubernetes, CNCF Primary at site A, synchronous standby at site B, backups and promotion Patroni
ClickHouse Columnar analytical database Ledger archive, raw payloads, noise, cost and interrupt reporting Apache Druid, OpenSearch
Ceph RGW S3-compatible object gateway for Ceph Object Lock bucket for hash-chain anchors and archived event-log partitions Garage
Kubernetes Container orchestrator, CNCF Platform-owned cluster for control-plane services only systemd hosts as for the paging path
Argo CD GitOps delivery for Kubernetes, CNCF Control-plane deployment and the site-B warm rebuild Flux
Keycloak Identity and access management, CNCF OIDC broker to the corporate identity provider, break-glass WebAuthn accounts, team scopes Authentik, Dex
OpenBao Secrets management, Linux Foundation fork of Vault Integration and carrier secrets, transit keys for contact data, platform PKI HashiCorp Vault (BUSL)
Mattermost Team chat, Team Edition Incident channels created by the chat bridge Matrix with Element, Zulip
Forgejo Git forge with CI Source, reviews and Forgejo Actions builds for platform code, catalogue and policies GitLab Community Edition
Harbor OCI registry, CNCF Signed binaries, images and SBOMs staged onto hosts before a release Zot
cosign Artefact signing, Sigstore Signs every paging-path artefact; verified by Ansible before staging Notation
Ansible Configuration management Builds and releases paging hosts, one cell and one host at a time Salt
chrony NTP implementation Time discipline for paging hosts against cell-local sources ntpd
Prometheus and Alertmanager Metrics and alert routing, CNCF Per-cell metrics, freshness signals and the platform's own alerts VictoriaMetrics
Loki Log aggregation Logs from paging hosts and control-plane services OpenSearch
OpenTelemetry Collector Telemetry pipeline, CNCF Bounded, node-local collection of traces and metrics Vector
Grafana Dashboards Operational dashboards and report views over Prometheus, Loki and ClickHouse Apache Superset
nginx Web server Static status page served from site C Caddy
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises34 views · HTML / SVG / draw.io