Architecture One-Pager

Incident Management Platform · Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises

A paging path that can wake the right person with the incident store, the console, the identity provider and both estate data centres gone, and a record that tells the truth about what happened afterwards.

Four hundred engineers are on call for forty services. When something breaks, a signal has to become a phone ringing in the right pocket within seconds, and it has to keep ringing, in more pockets, until somebody says they have it. That is a small system by traffic. It is a hard system by circumstance: it is used precisely when other systems are failing, and very often when the systems it runs on are the ones failing. A platform built like an ordinary internal application, one database, the shared Kubernetes cluster, corporate single sign-on, will work every day except the days it exists for. The second problem is trust. A platform that pages for noise teaches people to ignore it, and a platform that suppresses noise cleverly will one day suppress the real page. Everything below is arranged around those two facts.

The platform is two planes. The paging path receives alerts, decides whether to page, reads who is on call, runs the escalation, speaks to carriers and accepts acknowledgements. It runs as three paging cells, one in each estate data centre and one in a colocation outpost that shares neither their power nor their network. Each cell is three dedicated hosts running static Go services under systemd, HAProxy at the edge, and NATS JetStream holding the paging state, with a site-local domain for bulk ingest and a paging domain with one replica per site. Asterisk and Jasmin speak SIP and SMPP to two independent carriers, and a push relay speaks to Apple and Google. The control plane edits schedules and policies, runs the response, holds the record and produces reviews. It runs on a small platform-owned Kubernetes cluster at one site with a warm rebuild at the other. Schedules are resolved there, 72 hours ahead, verified by a second independent resolver, and published into the paging state as 15-minute coverage buckets. The incident timeline is an append-only PostgreSQL log with a hash chain; the console reads projections of it. The notification ledger is written by the dispatcher and archived in ClickHouse, which also holds raw payloads and the noise and cost reports. Keycloak brokers the corporate identity provider and holds break-glass accounts; OpenBao holds every secret and the per-responder keys that protect contact data.

What it is, and what it is not

• A paging path that reads a pre-resolved snapshot — not a schedule database, a console or an identity provider consulted while a page is being decided.
• Escalation that advances on a missing acknowledgement — not escalation that trusts a carrier's delivered receipt to mean a human saw the page.
• Duplicate pages accepted as the price of never missing one — not exactly-once delivery pursued at the cost of a page that was never sent.
• Suppression declared by a named person, with an expiry, always recorded — not dependency-aware suppression that decides on its own that a second failure is the same failure.
• An event log that is the record, with every view derived from it — not a mutable incident row whose history is whatever the last editor left.
• Three sites, one of them outside the estate, from the first release — not a single data centre with a disaster-recovery plan to add a second later.

The decisions that are the architecture

1. The paging path is a separate plane (ADR-01) — It holds a replicated copy of who is on call and how to escalate, and decides, dispatches and accepts acknowledgements with every other part of the platform unavailable. The rest of the design follows from this line.
2. Nothing it pages for is underneath it (ADR-02) — Paging cells run on dedicated hosts with no orchestrator, no shared DNS and no identity provider at run time, because the estate's Kubernetes, DNS and single sign-on are all things it must page about.
3. Quorum governs the record, never the page (ADR-04) — A cell cut off from the other two keeps paging from its last snapshot and labels what it did. Two pages for one problem is survivable; no page is not.
4. Coverage is resolved ahead and verified twice (ADR-14) — 72 hours of 15-minute buckets, published within 60 seconds of a change, checked by an independent second resolver before paging can read them. A DST night is computed three days early.
5. Only an acknowledgement stops escalation (ADR-16) — Receipts move a page to another route or channel inside a step. They never advance a step and never end one.
6. Carriers are aggregated, the last software hop is owned (ADR-20) — Asterisk and Jasmin on two carriers with separate contracts and networks, and a responder app of the platform's own, so a keypress, an SMS reply and a button are acknowledgements the platform receives itself.
7. The platform may decide not to page only in ways a human declared (ADR-12) — Maintenance windows with an owner and an expiry, and flood control that attaches rather than drops. SEV1 and human-declared incidents are never suppressed.
8. The log is the record (ADR-08) — Every alert, page, acknowledgement, role change and correction is an appended event with a hash chain. The review's intervals are computed from it, not typed in.

Why it holds up over time

An incident platform outlives several generations of monitoring tools, chat products, carriers and phones. The design keeps the decisions that encode organisational judgement, who is on call, what counts as an acknowledgement, when the platform may stay silent, in data and in written rules, and keeps every product behind a seam narrow enough to replace without touching those rules.

• The boundary is a rule, not a product. 'The paging path must work with everything else down' can be tested against any future component in one question: does this add something the paging path waits for? It survives a change of broker, of language, of carrier and of data centre.
• Carriers and chat are adapters. SIP, SMPP, APNs, FCM and SMTP are decades-old or platform-owned protocols. Changing carrier is a trunk and a route; changing chat product is one bridge. Neither touches escalation logic.
• Time is handled once, and checked twice. Every stored instant is UTC, every rotation is a named IANA zone, and two independent resolvers must agree. Zone rules will keep changing; governments do not consult on-call rotas. The design makes that a routine release.
• The record does not depend on today's schema. An append-only log of attributed events can be re-projected into whatever view a future console, analytics tool or regulator needs. A mutable record can only ever answer the questions its current columns anticipated.
• The bias is written down. Duplicate over miss, acknowledgement over receipt, declared over inferred suppression. Future teams under pressure to reduce noise or cost will meet these as stated principles with their reasons, not as accidents of an old implementation.
• Independence has a price on it. The outpost costs a known amount a year and is reviewed as insurance. A cost that is stated survives budget reviews; a cost that is discovered gets cut in the first one.
• Fallbacks are exercised, so they stay real. Cell loss, control-plane rebuild, partition, carrier loss, exhausted escalation and the manual fallback kit are drilled on a calendar. An untested fallback quietly stops working as the system around it changes.
• The platform measures what it costs people. Interrupts per responder and actionable page rate are reported every week. A platform that makes its own human cost visible keeps its users; one that does not is replaced by whatever they route around it with.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.

Scope

In scope

• Alert ingest over webhook and email, normalisation, deduplication, quarantine and raw payload retention
• Incident state machine, severity model, grouping, merge, split and human declaration
• Layered schedules, overrides, timezone-correct resolution, coverage snapshots and gap detection
• Escalation policies, push, SMS, voice and email across two carriers, and acknowledgement from every channel
• Maintenance windows, flood control, noise reporting and every suppression decision recorded
• Incident channels, roles, stakeholder updates and the service catalogue as the source of ownership
• Post-incident reviews with computed intervals, action items synchronised with the tracker, and cross-incident search
• The append-only event log, the notification ledger, the platform API and the outbound event stream

Explicitly out of scope

• Metric evaluation and alert rule authoring, which stay in the monitoring systems
• The chat product and the issue tracker themselves; the platform integrates with both
• A customer-facing status page beyond the static page the outpost serves
• Carrier networks and Apple and Google push services, which no on-premises design can own
• Any managed cloud service; every component runs on hardware the organisation owns

Prove it before production depends on it

Eight weeks, three real locations, two carrier contracts on trial terms, and four volunteer rotations. Each item replaces an assumption in this record with a measurement.

1. Measure JetStream key-value commit latency at p50 and p99 with one replica per site, under the storm replay, and with site C degraded to 30 ms
2. Kill the owner of a timer shard mid-escalation and measure fire accuracy across 1,000 takeovers
3. Send 10,000 SMS and 2,000 calls per carrier to the responders' real numbering ranges and measure receipt fidelity, DTMF recognition and time to ring
4. Apply for the Critical Alerts entitlement in week one, and test the voice-bypass fallback on iOS and Android in parallel
5. Partition site C for an hour during synthetic load and review every duplicate the reconciler labels
6. Rebuild projections from a synthetic seven-year log and time it
7. Resolve every rotation across every DST transition to 2030 with both resolvers and review every disagreement

• Pass: synthetic pages from all three cells at p95 under 15 s for the final four weeks, including during the storm replay.
• Pass: no timer fires more than 5 s late across the takeover test, and none fails to fire.
• Fail and rethink: either carrier's receipts arrive for fewer than 90% of delivered test messages, which moves the silence threshold from receipts to acknowledgement time alone.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 32 records across 8 areas, each with the alternatives that lost and what the choice costs.