# Incident Management Platform

**Solution Architecture v1.0 · open source, fully on-premises · Reliability Architecture · 2026-09 · 34 views · 32 architecture decision records**

An incident management platform for one organisation: 400 responders, 40 services and 25 on-call rotations across three timezones. It receives alerts, decides whether a human must be woken, works out which human, reaches them by push, SMS, voice and email, and keeps escalating until someone acknowledges. Around that loop it keeps an immutable incident timeline, coordinates the response and produces the post-incident review. It is built from open-source software on hardware the organisation owns: HAProxy and Go services at the edge and on the paging path, NATS JetStream for paging state across three sites, Asterisk and Jasmin on two independent carrier routes, PostgreSQL for the append-only event log, ClickHouse for the notification ledger and reporting, and Keycloak and OpenBao for identity and secrets.

The platform is used when other systems are failing, often including the ones it runs on. The design is arranged around one rule: the paging path can decide, dispatch and accept an acknowledgement with the incident store, the console, the identity provider and both estate data centres unavailable.

The decisions that carry the design:

- **The paging path is a separate plane.** It holds a replicated copy of who is on call and how to escalate, and calls nothing in the control plane.
- **Nothing it pages for is underneath it.** Paging cells run as static binaries on dedicated hosts, with no dependency on the estate's Kubernetes, DNS or single sign-on.
- **Three cells, one outside the estate.** Sites A and B host the estate; site C is a colocation outpost with its own power, uplink and carrier. All three page.
- **Quorum governs the record, never the page.** A partitioned cell keeps paging from its last snapshot and labels what it did. Duplicates are accepted; silence is not.
- **Coverage is resolved ahead and verified twice.** 72 hours of 15-minute buckets, checked by an independent second resolver, read with one lookup.
- **Only an acknowledgement stops escalation.** Carrier receipts move a page to another route or channel inside a step; they never advance or end one.
- **Silence is declared by a person.** Maintenance windows have an owner and an expiry, flood control attaches rather than drops, and SEV1 and human-declared incidents are never suppressed.
- **The log is the record.** Every change is an appended, hash-chained event; the review's intervals are computed from it.

The architecture one-pager (including why the design holds up over time) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~12 min) and [docs/decision-record.md](docs/decision-record.md) (~96 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 34 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 32 decision records, the technology table and the package glossary as markdown |
| `specs/part-a..f.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## Rebuild

```bash
bash scripts/build.sh
```

The build needs Node 20+ and the `architecture-diagram-web` skill. Every view passes the geometry check (0 errors, 0 warnings) and the routing check (216 edges, 0 errors, 0 warnings). Every icon resolves (503 nodes, none unresolved or weak), and every relative link resolves.

## Notes on the requirement

- **Stack.** The requirement targets Google Cloud with a second, out-of-cloud notification provider. As requested, this package runs on-premises with open source. The two cloud regions become estate data centres A and B; the out-of-cloud provider becomes site C, a colocation outpost with its own carrier contract, uplink and power, running a full paging cell (ADR-03).
- **The last mile is not ownable.** Carriers and Apple and Google push services are external by nature. The design owns everything up to them and every acknowledgement coming back, and never relies on push alone for SEV1 or SEV2 (ADR-20).
- **Multi-site paging moved into the MVP.** The requirement places multi-region active-active paging in Phase 3. A first release that shares a failure domain with the estate fails the test the platform exists for, so three cells ship in the MVP (view 30).
- **Provider failover.** A literal failover to the other provider on the same channel after a latency threshold would break the 60-second duplicate rule whenever a route is slow. The design retries the same channel on hard failure and moves to the next channel on silence (ADR-21).
- **Coverage horizon.** The requirement asks for at least 60 minutes of materialised coverage. The design materialises 72 hours, so DST boundaries are verified days ahead and a long control-plane outage leaves paging correct (ADR-14).
- **Unproven behaviours.** Six behaviours must be confirmed on the chosen releases and contracts before production depends on them, including stretched JetStream commit latency, both carriers' receipt and DTMF fidelity, and Apple granting the Critical Alerts entitlement. They are listed in the decision record's status note and each has a proof-phase item in the one-pager.