Architecture Decision Record

Incident Management Platform · Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises

The argument these decisions serve is summarised in the Architecture One-Pager.

Thirty-two decisions make up this architecture. Everything else on the thirty-four views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the products, the team and the estate have changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Latencies, volumes and costs are the requirement's targets or planning estimates, and each is replaced by a measurement in the proof phase. Six behaviours are relied on and must be confirmed on the exact releases and contracts chosen before production depends on them: NATS JetStream key-value writes committing within 50 ms at p99 with one replica at each of three sites; JetStream's per-key TTL behaving as documented for the 24-hour deduplication and 60-second human-dedup buckets; both carriers returning SMS delivery receipts and passing DTMF reliably enough to measure, on the numbering ranges the responders use; Apple granting the Critical Alerts entitlement to the responder app; Asterisk and Jasmin sustaining the 6,000-notification, ten-minute major-incident burst on one cell with one host down; and PostgreSQL rebuilding the operational projections from seven years of events inside 60 minutes.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it works on-premise: The concrete mechanism: which package, configured how, on whose hardware.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

Planes and independence: What sits on the path that wakes a human, what it is allowed to depend on, where each plane runs, and what happens when the sites cannot see each other.

• ADR-01 · The paging path is a separate plane from the incident record
• ADR-02 · Paging cells run on dedicated hosts with no estate dependency at run time
• ADR-03 · Three paging cells active-active, one outside the estate; control plane single-site with a warm rebuild
• ADR-04 · Quorum governs the record, never the page
• ADR-05 · Build the paging core; adopt open source for everything around it

State and records: Where paging state lives, how bulk traffic is kept off the site links, what the system of record is, and where evidence and reporting data go.

• ADR-06 · NATS JetStream holds the paging path's state
• ADR-07 · Two JetStream domains: site-local ingest, three-site paging
• ADR-08 · The incident timeline is an append-only PostgreSQL log; the record is a projection
• ADR-09 · ClickHouse for the ledger archive, raw payloads and reporting; Ceph Object Lock for tamper evidence

Ingest and decide: How untrusted alerts become normalised signals, who groups them, when the platform may stay silent, and what severity means.

• ADR-10 · Ingest edge: per-integration quotas at HAProxy, verify before parse, CEL mappings, quarantine
• ADR-11 · Grouping is owned here, on explicit keys that never cross a service boundary
• ADR-12 · The platform may decide not to page only in ways a human declared, and always says why
• ADR-13 · Severity is declared, and it binds the response obligations

Schedules and escalation: Where on-call is resolved, how time zones are made boring, what escalation advances on, how timers survive a crash, and what a handoff does to a live page.

• ADR-14 · On-call is resolved ahead of time into 15-minute coverage buckets, 72 hours out
• ADR-15 · Timezones are resolved in IANA zones, checked by a second resolver, and gaps are found 14 days out
• ADR-16 · Escalation advances only on absence of acknowledgement; delivery failure moves within a step
• ADR-17 · Escalation timers are sharded in-memory wheels over durable timer records, taken over by lease
• ADR-18 · A handoff adds the incoming responder to a live page; it never removes the outgoing one

Notification: Whether to own delivery or aggregate it, how providers fail over, what counts as a duplicate, how every channel acknowledges, and what happens when nobody does.

• ADR-19 · Deduplicate at the human, not at the sender; prefer sending twice to not sending
• ADR-20 · Aggregate carriers, own the last software hop: Asterisk, Jasmin, a push relay and a responder app
• ADR-21 · Fail over on the same channel after a hard failure, to the next channel after silence
• ADR-22 · Every channel acknowledges with a signed, single-use token; no login, no DNS dependency
• ADR-23 · Escalation exhaustion is a terminal state, a repeating broadcast and an incident against the platform

Response and learning: How the response is coordinated around the incident record, and how a review is made the cheapest thing to finish.

• ADR-24 · Coordination happens around the record: channels, roles, updates and ownership as events
• ADR-25 · Reviews are generated from the log, blameless by template, with action items tracked both ways

Security and identity: Signing in when the identity provider is down, integrations that are never trusted, contact data that can be erased from an immutable system, and incidents only some people may read.

• ADR-26 · Keycloak brokers the corporate identity provider and holds break-glass accounts on hardware keys
• ADR-27 · Every integration is untrusted, independently authenticated and revocable without a deploy
• ADR-28 · Contact data is encrypted per responder, so offboarding erases it from an immutable system
• ADR-29 · Incidents are readable by default; sensitive ones are restricted by row-level security, and the restriction is audited

Operations and cost: How the platform proves it works from outside itself, how the paging path changes without becoming the outage, and how money and sleep are kept visible.

• ADR-30 · A synthetic page runs every minute, an outside watchdog pages on carrier B, and a manual fallback kit is drilled
• ADR-31 · The paging path changes one cell at a time, never during a SEV1 or SEV2, from artefacts already on the host
• ADR-32 · Money and sleep are metered per service and rotation; the independence premium is a stated number

Technology by capability

Every capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns; carriers and Apple and Google push are the only external services, and none of them is a single dependency.

The decisions, and the alternatives that lost

Planes and independence

What sits on the path that wakes a human, what it is allowed to depend on, where each plane runs, and what happens when the sites cannot see each other.

ADR-01 · The paging path is a separate plane from the incident record

Status: Accepted · Shown on views: 02, 08, 22

What must still work for a human to be woken when the rest of the platform is unavailable?

Context. The platform is used when other systems are failing and often when its own dependencies are part of the failure. The requirement asks for 99.99% on the path from alert to acknowledgement and 99.9% for the console, schedule editing and reviews. A single application with one database gives every capability the availability of its least reliable part, and the least reliable parts are the richest ones: the console, search, reviews and analytics.

Decision. Split the platform into two planes. The paging path ingests, decides, reads coverage, runs escalation, dispatches and accepts acknowledgements, using only a replicated copy of the two things it needs: who is on call, and how to escalate. The control plane edits schedules and policies, runs the response, holds the event log and produces reviews. The paging path publishes events to an outbox that the control plane reconciles later. Nothing in the paging path calls the control plane.

How it works on-premise. Paging services run on dedicated hosts in three paging cells (ADR-02, ADR-03) with NATS JetStream as their only shared state (ADR-06). The control plane runs on a platform-owned Kubernetes cluster. The only data flowing from control plane to paging path is coverage buckets and escalation policies written into JetStream key-value buckets; the only data flowing back is the outbox. A conformance test fails any paging-path build that opens a connection to a control-plane address.

What it buys

• The console, schedule editor, reviews and analytics can all be down with no effect on paging
• Each plane gets the engineering discipline its target deserves and no more
• The critical surface is small enough to test exhaustively, including its failure modes

What it costs

• Two representations of on-call state and a staleness window between them
• A class of bug where the page goes to whoever was on call a minute ago
• Reconciliation of paging events into the record is a component that must itself be correct

Choose differently when. When the organisation is small enough that one rotation covers everything and a person can be phoned by hand, a single well-run application with a tested manual fallback costs less and fails in simpler ways.

Why it holds up over time. This is a dependency rule, not a technology choice. The broker, the language and the carriers can all change and the rule still decides every design review: a proposal either adds something the paging path waits for, or it does not.

Lesson. Give the part that must not fail the smallest possible set of things it needs, a copy of each, and no way to ask for more.

ADR-02 · Paging cells run on dedicated hosts with no estate dependency at run time

Status: Accepted · Shown on views: 09, 26, 27

What infrastructure may the paging path stand on, given that it must page about that infrastructure failing?

Context. The organisation's Kubernetes clusters, DNS, identity provider, container registry, secrets service and network core are all things the platform will page about. Each is a reasonable dependency for an ordinary service and a correlated failure for this one. A paging service scheduled on the estate's Kubernetes cannot restart when the scheduler is the incident; one that resolves carrier hostnames through corporate DNS cannot page when DNS is the incident.

Decision. Run the paging services as static binaries under systemd on hosts dedicated to the paging path, three per cell. At run time they need only their local disk, the other cells over mTLS, carrier endpoints by IP address, and time from the cell's own sources. Artefacts, configuration and tzdata are staged on every host before a release switches to them.

How it works on-premise. Go services built with CGO disabled, installed by Ansible from signed OCI artefacts pulled from Harbor ahead of the switch and kept locally. HAProxy fronts each cell with an anycast address announced by BIRD over BGP with BFD. Each cell runs local caching resolvers with carrier and Apple and Google endpoints pinned, and chrony against a GPS-disciplined source at the outpost and two independent public pools elsewhere. Hosts are 16-core, 64 GB, 2 TB NVMe machines with disk encryption bound to the TPM.

What it buys

• A failure of the estate's orchestration, DNS, registry or identity cannot stop paging
• Restart and rollback are local operations: a symlink and a systemd restart
• The host inventory is small enough to rebuild from Ansible in under an hour

What it costs

• Nine hosts managed outside the organisation's standard platform, with their own patching
• No automatic rescheduling: a lost host stays lost until replaced, covered by the other two in its cell
• The platform team needs host-level operational skills the rest of the organisation may have handed to Kubernetes

Choose differently when. When the estate's orchestration runs in a failure domain genuinely separate from every service it hosts, and has been measured to be so through several major incidents, running the paging path there removes nine hosts from the platform team's care.

Why it holds up over time. The rule is 'the paging path does not stand on what it pages for', and it outlives any orchestrator. If the estate later moves to something other than Kubernetes, or the paging path moves to different hosts, the same test decides where it may run.

Lesson. A dependency that is reliable enough for everything else can still be exactly the wrong dependency for the system that reports its failures.

ADR-03 · Three paging cells active-active, one outside the estate; control plane single-site with a warm rebuild

Status: Accepted · Shown on views: 09, 10, 26

Where does each plane run, and how much redundancy does each one buy?

Context. The requirement names a correlated failure class: the region hosting both the estate and the platform. On-premises, the estate runs in two data centres, A and B, that share a network core, an identity provider and often an operations team. The paging path must survive losing both. The control plane has a 15-minute RTO and may be single-region. The requirement also says a hot second control plane buys nothing it asks for.

Decision. Run a paging cell in site A, one in site B and one in site C, a colocation outpost on a different power grid, upstream carrier and network path from A and B. All three cells accept alerts and page. Run the control plane at site A with a synchronous PostgreSQL standby and scaled-to-zero manifests at site B, rebuilt there within 15 minutes when A is lost.

How it works on-premise. Site C holds a full paging cell, carrier B's SIP trunk and SMPP bind, a GSM modem bank, the synthetic probes, the dead-man switch and the static status page. It connects to A and B over the organisation's WAN and, independently, over its own internet uplink with an IPsec tunnel, so losing the WAN does not isolate it. The control plane's site-B rebuild is an Argo CD sync of the same manifests plus promotion of the CloudNativePG replica, timed quarterly.

What it buys

• The event that takes out both estate data centres is covered by a cell that has been paging all along
• Site C is also the natural home for everything that must observe the platform from outside
• The control plane stays simple enough to restore by a small team under pressure

What it costs

• A stated yearly premium for site C (ADR-32)
• Three cells to keep in step through releases, ordered and gated (ADR-31)
• The console is unavailable for up to 15 minutes if site A is lost, with cell status pages in the meantime

Choose differently when. An organisation whose two data centres are genuinely independent, different regions, carriers, power and operations teams, with no shared network core or identity, can treat B as the outside cell and save the outpost.

Why it holds up over time. The shape, some cells inside the estate and at least one outside it, holds whatever the estate becomes. If the estate moves to a cloud, the outside cell stays on-premises; if it stays on-premises, the outside cell can move to a different provider. The rule is that one cell never shares the estate's fate.

Lesson. Redundancy only counts against the failures it does not share. Count the shared failures before counting the copies.

ADR-04 · Quorum governs the record, never the page

Status: Accepted · Shown on views: 12, 23, 34

What does a paging cell do when it cannot reach a majority of the paging domain?

Context. The paging domain uses Raft with one replica per site. A cell partitioned from the other two cannot commit writes: it cannot open an incident with a compare-and-set, write a timer record or record an acknowledgement durably. The conventional answer for a replicated system is to stop writing and serve nothing. For this system the partitioned cell may be the only one that can still see a failing estate and still reach a carrier.

Decision. A cell that loses quorum enters minority mode. It continues to decide and page using its last coverage snapshot and policies, keeps escalation state and events in a local append-only log, accepts acknowledgements it can receive, marks every message it sends as coming from a partitioned cell, and pages the platform rotation about its own state. When quorum returns, the local log is replayed through the reconciler, which links and labels duplicate incidents rather than merging them.

How it works on-premise. Each paging service watches its JetStream connection's cluster state and the paging domain's meta-leader. After 10 seconds without quorum it switches to minority mode, writing to a local file-backed log on NVMe with fsync per event. Coverage buckets are read from the local replica, which JetStream keeps serving for direct gets. The reconciler replays minority logs idempotently by event id and writes partition-duplicate links as events.

What it buys

• Every partition shape leaves at least one cell able to page whoever it can reach
• The bias toward duplicates is implemented, not just stated
• The review can see exactly what each side of a partition did

What it costs

• A responder may be paged by both sides of a partition, and called again after acknowledging on the other side
• Minority mode is a second code path and must be exercised by the quarterly partition drill
• The reconciler must handle incidents that exist twice for a while

Choose differently when. If duplicate pages carried real harm, for example a page that dispatches a physical crew, the balance changes and a minority cell should page only a small, explicitly configured set of escalation targets.

Why it holds up over time. The principle is independent of Raft or JetStream: consistency protocols decide what a system may record, and a paging system must not let them decide whether it speaks. Any future replacement for the paging state inherits the same split.

Lesson. Decide separately what your consistency model protects and what it is allowed to prevent. They are not always the same thing.

ADR-05 · Build the paging core; adopt open source for everything around it

Status: Accepted · Shown on views: 08, 30

Is there an open-source incident platform to adopt, or does the core need to be built?

Context. The requirement is unusually specific about the paging path: a separate plane, ahead-of-time coverage, acknowledgement-driven escalation with durable timers, provider failover inside a step, human-side deduplication, minority operation and an event-sourced record. Open-source options exist. GoAlert, from Target, is mature and well designed for its purpose, with PostgreSQL as its single store and Twilio for SMS and voice. Grafana OnCall's open-source edition was put into maintenance mode in 2025 with archival announced. Netflix Dispatch coordinates response and leaves paging to an external on-call service.

Decision. Build five paging services (ingest gateway, incident engine, escalation timers, dispatcher, ack receiver) and the control-plane domain services (schedule resolver, incident service, review module, console). Adopt open source for every other part: edge, messaging, databases, voice, SMS, identity, secrets, chat, observability and delivery. Keep GoAlert as the named fallback if the team cannot sustain the built core.

How it works on-premise. The built services are Go, with a TypeScript console. Each paging service has a written contract, a property-based test suite and a conformance suite run against every release: timer accuracy across restarts, acknowledgement cancellation latency, human-dedup behaviour under failover, and minority mode. The built surface is sized at roughly 30,000 lines for the paging path, owned by a team of six.

What it buys

• The paging path is exactly what the requirement asks for, with no inherited assumption to work around
• Every adopted component has a large user base and published failure behaviour
• The team's engineering effort goes to the part that differentiates reliability

What it costs

• A team must own an escalation engine, timers and a dispatcher indefinitely
• Bugs in the built core have no upstream community to find them first
• Feature requests common in commercial tools, such as rich mobile apps, arrive slowly

Choose differently when. An organisation of a few dozen responders on one site, allowed to use a cloud SMS and voice provider, should run GoAlert. Most of this design's complexity pays for independence and scale that organisation does not need.

Why it holds up over time. The build boundary sits at the requirement, not at a product. If a future open-source project implements a separate paging plane with ahead-of-time coverage, the built core can be retired behind the same event and coverage contracts without touching the control plane or the record.

Lesson. Build the part whose requirements you would have to fight a product to meet, and nothing else.

State and records

Where paging state lives, how bulk traffic is kept off the site links, what the system of record is, and where evidence and reporting data go.

ADR-06 · NATS JetStream holds the paging path's state

Status: Accepted · Shown on views: 08, 09, 14

What durable, replicated store and log does the paging path share across cells?

Context. The paging path needs a small amount of strongly consistent state (open incidents by grouping key, escalation state, timer records, acknowledgements, coverage buckets, policies), a durable ordered log for the outbox and the ledger, key-value entries with per-key expiry for deduplication, and a site-local buffer for bulk ingest. It must replicate across three sites, run on nine hosts without an orchestrator, and be operable at 03:00 by the same team that runs the rest of the path.

Decision. Use NATS JetStream for all of it: streams for ingest, outbox and ledger; key-value buckets for coverage, policies, incident state, timers and deduplication. One binary, one operational model, one client library in every paging service.

How it works on-premise. nats-server on every paging host, configured as two JetStream domains (ADR-07). Key-value buckets use three replicas with one per site; streams for ingest use three replicas within a cell. Per-key TTL is used for the 24-hour alert dedup and the 60-second human dedup. Accounts separate ingest, paging and control-plane publishers, and every connection uses mTLS from the platform PKI.

What it buys

• All paging state has one replication and failure model to understand and drill
• Compare-and-set on grouping keys and incident state gives exactly-one incident while sites can talk
• The footprint is small enough for three hosts per cell to carry comfortably

What it costs

• The team must learn JetStream's Raft behaviour, placement and recovery in depth
• Long-retention analytics cannot live in JetStream and are sunk to ClickHouse
• A defect in JetStream affects the whole paging path at once, which is mitigated only by staged upgrades one cell at a time

Choose differently when. If the paging path grew to handle millions of events a second with long retention, or the organisation already ran Kafka as a mature internal service outside the estate's failure domain, Kafka for streams plus a separate consistent store would be the better split.

Why it holds up over time. The paging services depend on four operations: publish to a stream, consume from a stream, compare-and-set a key, get a key. Any future store offering those four with replication can replace JetStream behind a thin internal interface, and the contract tests already exercise exactly those four.

Lesson. When a critical path needs a few kinds of state, one well-understood store that provides all of them beats the best store for each.

ADR-07 · Two JetStream domains: site-local ingest, three-site paging

Status: Accepted · Shown on views: 09, 16, 21

How is a 20,000-alert-a-second storm kept from competing with paging state for the links between sites?

Context. Replicating every alert across three sites would push storm traffic, 2.4 million messages with payloads, through the Raft groups that also carry acknowledgements and timer writes. Site C's link is the slowest and least predictable. Paging state writes are few, a handful per incident, and must be replicated across sites. Alert writes are many and need durability within a site only, because an alert is reproducible by its source until acknowledged with a 202.

Decision. Run two JetStream domains on the same hosts. The ingest domain is local to each cell, with three replicas across that cell's hosts, and carries raw and normalised alerts. The paging domain has one replica per site and carries incident state, timers, acknowledgements, coverage, policies, the outbox and the ledger. Only the incident engine crosses from one to the other, and only with decisions.

How it works on-premise. Separate nats-server processes and ports per domain on each host, with separate storage directories on the NVMe and separate CPU weights in systemd. Ingest streams are limited to 7 days or 400 GB per cell. The paging domain uses placement tags so each replica lands in a different site. Commit latency on the paging domain is measured continuously and alarmed at 50 ms p99.

What it buys

• Storm volume is bounded by one cell's disks and CPU, not by the WAN
• Paging domain commit latency stays predictable during storms
• A cell can be drained of ingest without touching paging state

What it costs

• Alerts admitted by a cell that is then lost before the engine consumes them must be re-sent by their source, which the 202 contract makes explicit
• Two domains per host to configure, monitor and upgrade
• Deduplication is site-local, so a retry that lands on another cell attaches as a counted duplicate rather than being dropped

Choose differently when. If sites were connected by low-latency, high-capacity links with no meaningful cost or congestion, one stretched domain would be simpler and would make deduplication global.

Why it holds up over time. The separation is by the nature of the data, reproducible bulk signals versus irreplaceable decisions, and would hold on any messaging technology. It is also the seam along which ingest could move to a different store without touching paging state.

Lesson. Replicate decisions widely and signals locally. They have different volumes, different value and different owners of the retry.

ADR-08 · The incident timeline is an append-only PostgreSQL log; the record is a projection

Status: Accepted · Shown on views: 14, 15, 18

Is the incident record the source of truth, or a projection of an event log?

Context. The review, the audit obligation and the merge and split operations all need to answer questions about the past: what was believed at 03:14, who changed severity and why, what the incident looked like before two were merged. A mutable incident row answers only the present. Retention is seven years for timelines and audit events, which must be immutable. Volume is modest: 5,000 incidents a month and on the order of tens of millions of events a year including attached alert summaries.

Decision. Store every change as an appended event in PostgreSQL: alerts attached, state transitions, severity changes, pages, acknowledgements, role changes, notes, merges, splits, corrections. Each event has an idempotent id, a per-incident sequence, the platform receive time, an actor and the hash of the previous event in that incident. Incident views are projections rebuilt from the log. Corrections are new events that supersede earlier ones.

How it works on-premise. PostgreSQL 17 on CloudNativePG, primary at site A, synchronous standby at site B, one asynchronous replica for reporting. The event table is partitioned monthly. Application roles have INSERT and SELECT only, and a trigger rejects UPDATE and DELETE for every role including the owner. A daily job writes each incident's chain head and a partition checksum to a Ceph RGW bucket with Object Lock in compliance mode. Closed partitions older than two years are exported to Parquet in the same bucket and kept queryable.

What it buys

• The review's intervals and responders are computed from facts, not typed from memory
• Merge, split and correction preserve every earlier state
• Tampering with history is detectable from outside the database

What it costs

• Projection rebuild is an operational concern that must be drilled
• Queries about current state go through projections, which can lag the log
• Schema evolution of event payloads needs versioned readers rather than migrations

Choose differently when. For a platform that never merges or corrects incidents and has no audit obligation, a mutable record with a change log is cheaper to build and to query, and loses little.

Why it holds up over time. An attributed, ordered, append-only log can be re-projected into any future console, analytics model or regulator's question. The storage engine can change; the event contract, id, sequence, time, actor, type, payload, previous hash, is what carries seven years of history forward.

Lesson. If the questions you will be asked are about the past, store the past. A current state can always be derived; a lost history cannot.

ADR-09 · ClickHouse for the ledger archive, raw payloads and reporting; Ceph Object Lock for tamper evidence

Status: Accepted · Shown on views: 14, 16, 29

Where do high-volume evidence and reporting data live once they leave the paging path?

Context. The notification ledger must be retained for 13 months and queried for cost, delivery fidelity and review evidence. Raw alert payloads must be kept for 90 days and searched when a mapping is wrong. The weekly noise and cost report aggregates alerts, incidents and attempts per service. These are append-heavy, scan-heavy workloads that would crowd the event log's database, and none of them may slow down the console.

Decision. Sink the ledger stream, raw payloads and alert metadata from JetStream into ClickHouse with table TTLs matching each retention. Build the noise, cost and interrupt reports as materialised views there. Keep hash-chain anchors and archived event-log partitions in a Ceph RGW bucket with Object Lock.

How it works on-premise. ClickHouse with two replicas at site A and one at site B, coordinated by ClickHouse Keeper, fed by consumers that commit JetStream acknowledgements only after a successful insert. Payloads use ZSTD compression. Grafana reads ClickHouse for dashboards; the weekly report is rendered by a job and emailed. Ceph RGW runs on the organisation's existing Ceph cluster with a dedicated pool and a compliance-mode Object Lock retention of seven years.

What it buys

• Cost per page, per incident and per service is a query that returns in seconds
• A wrong mapping can be diagnosed and re-run against 90 days of raw payloads
• The event log's database carries only the record

What it costs

• One more database technology for the team
• Reports can lag the ledger by the sink interval, which the degradation order allows
• ClickHouse's replication and merges need their own monitoring

Choose differently when. If payload search became the dominant workload, with free-text queries across months of vendor emails, OpenSearch for payloads beside ClickHouse for aggregates would be worth its extra footprint.

Why it holds up over time. The data leaves the paging path as streams with stable schemas. Any future analytical store can be fed from the same streams, and the Object Lock archive in Parquet is readable by most analytical engines that will exist in ten years.

Lesson. Keep the record's database for the record. Evidence and reporting grow faster and are queried differently.

Ingest and decide

How untrusted alerts become normalised signals, who groups them, when the platform may stay silent, and what severity means.

ADR-10 · Ingest edge: per-integration quotas at HAProxy, verify before parse, CEL mappings, quarantine

Status: Accepted · Shown on views: 12, 16, 21

How are heterogeneous, untrusted alert sources admitted and normalised without one of them degrading the others?

Context. Alerts arrive from metric and log alerting, synthetics, vendor health feeds, ticketing systems and email. Integrations will send malformed payloads, retry aggressively, flood during their own incidents and occasionally be hostile. The requirement asks for one normalised schema, the raw payload kept, per-integration quotas that shed only the offender, and a quarantine path that raises an incident against the integration owner rather than failing silently.

Decision. Admit at HAProxy with a per-integration rate limit keyed by credential. In the gateway, verify the HMAC signature and replay window before parsing, then evaluate the integration's CEL mapping in a sandbox with time and memory limits. Valid alerts are published with the raw payload in one message; invalid ones are published to quarantine with the reason. A quarantine monitor opens a SEV4 incident against the integration owner when its quarantine rate crosses a threshold.

How it works on-premise. HAProxy stick tables track request rate per integration id extracted from a header, returning 429 with Retry-After above the quota. The gateway is Go, using cel-go with a cost limit per evaluation. Mappings are versioned objects edited in the console by integration owners, published to the paging domain like coverage. Postfix accepts mail only for per-integration addresses from allowlisted senders and hands parsed messages to the same gateway path.

What it buys

• One noisy or broken integration costs only itself
• Integration owners fix their own mappings, and can re-normalise past payloads
• Quarantine volume is visible as an incident, not a log line

What it costs

• CEL is one more language for integration owners to learn
• Quotas need tuning per integration and review after each storm
• Accepting malformed payloads with 202 means a broken integration can go unnoticed until the quarantine threshold

Choose differently when. A small, fixed set of sources all owned by one team can skip mappings and quotas and agree a schema directly; the machinery pays for itself when integrations are many and owned by different people.

Why it holds up over time. The normalised alert schema and the quarantine contract are the stable part. Sources, transports and even the expression language behind mappings can change without affecting anything after the ingest domain.

Lesson. Refuse cheaply, before understanding. Keep what you could not understand, and tell its owner.

ADR-11 · Grouping is owned here, on explicit keys that never cross a service boundary

Status: Accepted · Shown on views: 05, 12, 13

Who owns grouping: the monitoring systems, or this platform?

Context. Upstream alerting knows topology and intent; Alertmanager already groups by labels. This platform sees every source at once but does not own a dependency model. The organisation's worse error is the subtle one: a second, unrelated failure attached to an incident someone has already acknowledged, so nobody is paged for it. An alert storm that buries the signal is loud and gets fixed; over-grouping is quiet.

Decision. Group on an explicit grouping key: by default service, environment and the source's own group key when it provides one, otherwise the fingerprint family. Honour upstream grouping as an input, never replace it. An alert for a different service never joins an open incident automatically. An attached alert with a higher proposed severity, or from a service not yet in the incident, re-pages. Responders can merge, split and detach, as events that preserve both timelines.

How it works on-premise. The grouping key is computed in the incident engine from the normalised alert and the service catalogue entry, and claimed with a compare-and-set in the paging domain's open-incidents bucket. Alertmanager's groupKey passes through normalisation as source_group. Merge and split are control-plane commands that append events and update the open-incidents bucket through the same compare-and-set.

What it buys

• A responder can always explain why two alerts are in one incident
• A cascade across services produces one page per affected service, not one per alert, and not one in total
• History survives every regrouping a responder performs

What it costs

• Cascading failures page several rotations, each of which may see the same root cause
• Grouping quality depends on the service catalogue being right
• Merge and split are manual work during the worst incidents

Choose differently when. In an organisation where storms routinely page a dozen rotations for one upstream cause and the dependency graph is generated from real traffic rather than declared, topology-aware grouping earns its risk.

Why it holds up over time. The key, service boundary rule and re-page conditions are policy, stored as data. Better correlation can be added later as annotation without changing what is allowed to suppress a page, which is the part that must not drift.

Lesson. Choose the grouping error you can see. A storm is obvious; a hidden second failure is not.

ADR-12 · The platform may decide not to page only in ways a human declared, and always says why

Status: Accepted · Shown on views: 13, 21, 34

Where may suppression be automatic rather than declared in advance by a person?

Context. Maintenance windows, flood control and dependency-aware suppression all reduce noise, and each is a mechanism for missing a real incident. Paging for everything destroys the rotation and then trust. The requirement caps suppression at 30 days, requires owner and expiry, forbids suppressing SEV1 or human-declared incidents, and requires every suppression decision to be logged with its rule.

Decision. Evaluate five checks in a fixed order: exact duplicate, exemption (SEV1 or human-declared), declared maintenance window, attachment to an open incident, and flood control per grouping key. The only automatic silences are exact duplicates and attachment, and both record the alert. Maintenance windows must name an owner and expire within 30 days, renewed explicitly. Flood control attaches and counts; it never drops. Dependency knowledge annotates incidents and never withholds a page. Every decision writes an event naming the rule.

How it works on-premise. Suppression rules are control-plane objects, restricted to a narrower role than schedule editing, published into the paging domain as versioned policy. The engine evaluates them in memory against the published version and stamps the version on every decision event. A daily job lists rules expiring within seven days to their owners; expired rules stop matching at their expiry instant with no grace period.

What it buys

• A review can establish in one query whether the platform chose not to page, and on whose rule
• No suppression outlives the memory of why it was created
• SEV1 and human-declared incidents are immune to every configuration mistake

What it costs

• More pages during cascading failures than a clever suppressor would send
• Owners must renew long maintenance periods monthly
• Noise reduction depends on people fixing alert rules, which the weekly report pushes but cannot force

Choose differently when. Where alert volume from cascades is so high that people are demonstrably ignoring pages, measured by acknowledge-and-resolve-with-no-action rates, a narrowly scoped automatic suppression with a strict audit may be the lesser harm.

Why it holds up over time. The ordering and the rule 'annotate, never withhold' are written principles, not code accidents. Future correlation technology can be as clever as it likes on the annotation side without being allowed to change who is woken.

Lesson. A suppressed real page costs more trust than a hundred false ones. Make silence something a person signs for.

ADR-13 · Severity is declared, and it binds the response obligations

Status: Accepted · Shown on views: 05, 19, 29

Is severity a computed property of alerts or a declared property of an incident, and what does it control?

Context. Alerting rules propose a severity, and they are often wrong in both directions. Responders learn more in the first minutes than any rule knew. The requirement asks for four levels, a proposed default, any responder able to raise or lower it with an attributed reason, and obligations bound to severity: acknowledgement target, channel and commander creation, stakeholder notification, mandatory review.

Decision. Severity is a declared field on the incident, initialised from the highest proposed severity among its alerts and changed only by an attributed event with a reason. Each level maps to an obligations profile: acknowledgement target and escalation step length, channel policy, whether a Mattermost channel and incident commander role are created, the stakeholder update cadence and whether a review is mandatory. Changing severity re-evaluates the profile immediately.

How it works on-premise. Obligation profiles are versioned control-plane objects published to the paging domain alongside escalation policies, so the dispatcher applies SEV1 channel policy even when the control plane is down. The incident service applies the non-paging obligations (channel, roles, stakeholder cadence, review) when it reconciles the severity event. Email-sourced alerts cannot propose SEV1.

What it buys

• A responder raising severity at 03:15 gets a channel, a commander role and a stakeholder clock without asking
• Every severity change and its reason is in the timeline for the review
• SEV1 paging behaviour does not depend on the control plane

What it costs

• Severity inflation by cautious responders produces more channels and reviews
• Obligation profiles are policy that someone must own and revisit
• Downgrading severity mid-incident stops obligations that people may have been relying on, so the change is announced in the channel

Choose differently when. In a highly automated environment where severity maps directly to measured customer impact, such as error-budget burn per service, computed severity with a human override may be more consistent.

Why it holds up over time. Four levels and a profile per level are an organisational contract that outlasts tools. The profiles are data, so changing what SEV2 means next year is an edit, not a release.

Lesson. Let signals propose and people decide, and make the decision do the paperwork.

Schedules and escalation

Where on-call is resolved, how time zones are made boring, what escalation advances on, how timers survive a crash, and what a handoff does to a live page.

ADR-14 · On-call is resolved ahead of time into 15-minute coverage buckets, 72 hours out

Status: Accepted · Shown on views: 02, 06, 17

Where is the on-call schedule resolved: at page time, or ahead of time?

Context. Page-time resolution is always current but puts rotation layering, overrides, timezone rules and the schedule database inside the paging path. A pre-materialised snapshot makes paging a single read against a replicated structure, at the cost of a staleness window and a second thing that can be wrong. The requirement asks for at least 60 minutes of materialised coverage, O(1) lookup regardless of layering, and staleness under 60 seconds after a change.

Decision. Resolve schedules in the control plane into coverage buckets keyed by rotation and 15-minute UTC interval, each holding the ordered layers and the verified contact methods for that interval. Materialise 72 hours ahead, re-resolve only the affected rotation on any change, verify with an independent resolver (ADR-15), and publish with a version into the paging domain. The dispatcher reads one bucket. If buckets run out, the last available bucket keeps serving and the platform rotation is paged about staleness.

How it works on-premise. The schedule resolver is a Go service reading the schedule store in PostgreSQL. Buckets are JetStream key-value entries named coverage.., written with quorum, around 7,000 live entries across 25 rotations. Escalation policies and obligation profiles are published the same way. Change events from the schedule editor trigger resolution within a second; a full re-resolution runs every 15 minutes as a safety net.

What it buys

• A deeply layered rotation pages as fast as a flat one
• Coverage for any past page is reproducible from the snapshot version stamped on it
• DST boundaries and gaps are resolved and checked days before they matter

What it costs

• A change can take up to 60 seconds to reach paging, so the outgoing responder may get one more page
• Two representations of coverage to keep consistent and monitor
• Contact data is replicated into the paging path, encrypted (ADR-28)

Choose differently when. A handful of flat rotations in one timezone, with a schedule store that is itself outside the estate's failure domain, does not need materialisation; the complexity pays for layering, timezones and independence.

Why it holds up over time. The bucket is a contract: rotation, interval, ordered targets, version. Scheduling rules can grow new layer types, fairness algorithms or calendar integrations, and none of that reaches the paging path, which will read the same bucket in ten years.

Lesson. Compute the hard answer early, check it, and hand the critical path a lookup.

ADR-15 · Timezones are resolved in IANA zones, checked by a second resolver, and gaps are found 14 days out

Status: Accepted · Shown on views: 06, 17, 24

How does the platform avoid an uncovered hour or a double page at a daylight-saving transition?

Context. Twenty-five rotations across three timezones hand over in local time, and twice a year some handoff times do not exist or happen twice. Timezone rules change with little notice when governments decide. Schedule bugs are silent until the hour they affect, which is usually a night. The requirement asks for DST-correct resolution tested against boundaries, gap detection 14 days ahead, and snapshots verified before the boundary.

Decision. Define rotations in local wall-clock time in a named IANA zone and store every materialised instant in UTC. Specify the rules for non-existent local times (next valid instant) and repeated ones (first occurrence). Implement resolution twice, independently, and publish buckets only when both agree; on disagreement, hold the previous verified buckets and notify the rotation owner. Run gap detection hourly over 14 days. Pin tzdata per release and roll updates out as releases verified the same way.

How it works on-premise. The primary resolver is Go using its embedded time zone database. The verifier is a separate implementation in Python using zoneinfo, written by a different engineer from the written rules rather than from the Go code, and run as its own service. The release pipeline resolves every rotation across every transition in its zone to 2030 with both. Gap warnings go to the rotation owner by email and chat; gaps inside 24 hours also open a SEV4 against the rotation.

What it buys

• A handoff at a time that does not exist is found days early, with both answers shown
• Zone rule changes from upstream become an ordinary release
• Coverage gaps reach the person who can fix them before they reach a page

What it costs

• Two implementations of the most intricate logic in the control plane
• A disagreement freezes publication for that rotation until someone looks
• A Python service in an otherwise Go control plane

Choose differently when. Rotations all in zones without DST, with handoffs at fixed UTC-friendly hours, get almost nothing from the second resolver and can drop it.

Why it holds up over time. Time rules will keep changing because they are political. Written resolution rules and two independent implementations turn every future change into a disagreement that is visible, instead of a page that silently goes to nobody.

Lesson. For logic that is easy to get subtly wrong and rarely exercised, two independent implementations that must agree are worth more than one well-tested one.

ADR-16 · Escalation advances only on absence of acknowledgement; delivery failure moves within a step

Status: Accepted · Shown on views: 19, 20, 25

What does escalation advance on: absence of acknowledgement, or delivery failure?

Context. Non-acknowledgement is the honest signal but waits the whole step interval even when the platform knows delivery failed. Advancing on delivery failure is faster and wrong whenever a receipt is wrong, and receipt fidelity varies greatly: push acceptance by Apple says nothing about the phone, some SMS carriers report delivered on hand-off, voice ring and answer are reliable but a voicemail answer is not a human. A premature escalation wakes a second person and teaches the first that their pager is optional.

Decision. Use two mechanisms with different powers. The step timer, and only the step timer, advances to the next escalation step, and only if no acknowledgement has been recorded. Within a step, delivery evidence drives a fast path: a hard failure retries the same channel on the other route; silence past the channel's threshold moves to the next channel in the step's channel policy; a target with no deliverable method at all falls immediately to the next target. A coverage gap, no resolved responder at all, falls immediately to the policy's final step.

How it works on-premise. The dispatcher owns the within-step fast path using per-carrier thresholds from continuously measured route health (ADR-21). Escalation timers own step advancement (ADR-17). Voice acknowledgement requires a DTMF keypress after the summary is read, so a voicemail pickup is recorded as answered-not-acknowledged. Receipts and outcomes are written to the ledger and never read by the timer.

What it buys

• Known delivery failures are acted on within seconds, inside the same step
• Nobody is escalated past because a carrier misreported
• Step timing is predictable for responders and managers

What it costs

• A responder whose every channel is silently failing is not escalated past until the step ends
• Per-carrier thresholds need continuous measurement
• Voicemail pickups cost a full step of waiting unless another channel reaches the person

Choose differently when. Where the platform owns an end-to-end delivery receipt it trusts, such as an app-level read receipt from its own client, a step could safely advance early on a confirmed failure to that client specifically.

Why it holds up over time. 'Acknowledgement is the only proof' is a principle that survives every change of channel. New channels, a smartwatch, a pager network, a messaging app, slot into the within-step fast path with their own thresholds without touching what advances a step.

Lesson. Let evidence speed up what you do next, but only let proof decide that you are done.

ADR-17 · Escalation timers are sharded in-memory wheels over durable timer records, taken over by lease

Status: Accepted · Shown on views: 09, 19, 34

Where do escalation timers live, and how do they fire on time after the process holding them dies?

Context. Peak load is modest, 120 open incidents with several targets and channels each, but a single timer that does not fire is a page that does not escalate. Accuracy must be ± 5 s at p99, a timer must survive a process restart and a host loss, cancellation by acknowledgement must take effect within 5 s, and a duplicate fire after failover is acceptable under the duplicate-over-miss principle. The candidates are an in-memory wheel with a write-ahead log, scheduled messages on a durable queue, or rows polled by a scanner.

Decision. Store every timer as a durable record in the paging domain before it is armed. Partition timers into 64 shards by incident id. Each shard is owned by one timer process through a lease; the owner loads the shard's records into a one-second timer wheel and fires from memory. Before firing it re-reads the incident state and skips cancelled or acknowledged work. When an owner stops renewing, another process takes the lease and rebuilds the wheel from the records. Fired timers are marked done with a compare-and-set; a timer that was fired but not marked may fire again.

How it works on-premise. Timer records and shard leases are JetStream key-value entries in the paging domain. Leases are renewed every second and expire after three seconds. Every paging host runs a timer process and competes for shards, so shards spread across all three cells; the process prefers shards whose incidents belong to its own cell's grouping claims. Rebuilding a shard from records takes well under a second at the expected volume. In minority mode, a cell's timer processes keep firing the shards they hold and adopt orphaned shards from their local copy.

What it buys

• Timers fire within about a second of due in normal operation, and within the lease expiry plus rebuild after a failure
• Adding hosts spreads shards without any repartitioning of data
• Cancellation is a state check at fire time, so an acknowledgement stops work already queued

What it costs

• A timer may fire twice around a takeover; the human-side dedup absorbs most of those
• Lease and shard logic is custom code on the most critical path, tested by 1,000-takeover runs in the proof phase
• Clock discipline on paging hosts is part of timer correctness

Choose differently when. If a future broker offered scheduled delivery with cancellation, replication and published accuracy, handing time to the broker would remove the most intricate custom code in the paging path.

Why it holds up over time. The durable record is the source of truth and the wheel is a cache. Either can be replaced independently: a different store for records, a different scheduling structure in memory. The contract, a timer exists before it is armed and state is checked before it fires, carries over.

Lesson. Keep time in memory for accuracy and on disk for truth, and design so that firing twice is harmless.

ADR-18 · A handoff adds the incoming responder to a live page; it never removes the outgoing one

Status: Accepted · Shown on views: 06, 19

What happens to an unacknowledged page when a shift ends in the middle of its escalation?

Context. Targets are resolved when a step fires. If the shift changes before acknowledgement, re-resolving would move the page to the incoming responder, who has no context and may not be settled in, and silence the outgoing responder, who may be about to acknowledge. The requirement forbids transferring a triggered page to the incoming responder without continuing to page the outgoing one.

Decision. When a step's targets change because coverage changed during an unacknowledged escalation, the escalation's target set becomes the union of the targets at trigger time and the current targets. Both are paged until anyone acknowledges. The handoff is recorded as an event on the incident. Acknowledged incidents are not re-targeted at handoff; ownership of an acknowledged incident moves only by explicit reassignment.

How it works on-premise. The escalation state in the paging domain stores the snapshot version and targets used at trigger. At each fire, the dispatcher reads the current bucket, computes the union, and records added targets as events. Human-side dedup keys include the responder, so the union never pages the same person twice on one channel within 60 seconds.

What it buys

• Shift boundaries cannot create an unpaged minute
• The timeline shows exactly who was added at the handoff and why
• Outgoing responders who are already working the page are not cut off

What it costs

• Two people are paged for incidents that span a handoff
• Rotations with frequent handoffs see slightly more interrupts in reporting
• Explicit reassignment is required for acknowledged incidents, which is one more action for the outgoing responder

Choose differently when. A rotation with a formal, attended handover meeting where live incidents are transferred verbally may prefer pinning to the outgoing responder and an explicit transfer step.

Why it holds up over time. The rule is about people at boundaries, not about any schedule model. New layer types, follow-the-sun rotations or automated shift swaps all meet the same rule at the moment coverage changes under a live page.

Lesson. At a boundary, add rather than replace. Removing someone from a live problem should always be a deliberate act.

Notification

Whether to own delivery or aggregate it, how providers fail over, what counts as a duplicate, how every channel acknowledges, and what happens when nobody does.

ADR-19 · Deduplicate at the human, not at the sender; prefer sending twice to not sending

Status: Accepted · Shown on views: 09, 12, 20

How does an at-least-once paging path avoid spamming a responder without risking a missed page?

Context. Every layer of the paging path retries: the gateway, the engine, the timers after takeover, the dispatcher across routes, and minority cells during partitions. Exactly-once delivery to a phone is not achievable across carriers, and trying for it at the sender means sometimes not sending. The requirement asks that a responder never receive the same page twice on the same channel within 60 seconds, and that the platform prefer sending twice over risking not sending.

Decision. Allow every component to retry freely. Immediately before a send, the dispatcher claims a key made of responder, channel, incident and step with a 60-second expiry. If the claim fails because the key exists, the send is skipped and recorded as a deduplicated attempt. If the claim cannot be made at all, because the paging domain is unavailable or the cell is in minority mode, the dispatcher sends anyway and records that deduplication was unavailable.

How it works on-premise. Human-dedup keys are JetStream key-value entries with a 60-second per-key TTL in the paging domain, created with create-if-absent semantics. In minority mode the dispatcher uses a local in-memory dedup table, which prevents duplicates within the cell but not across the partition. Every skipped and every unguarded send is a ledger entry with its reason.

What it buys

• Takeovers, failovers and replays do not need careful exactly-once handling
• The 60-second rule is enforced in one place and is testable there
• Every duplicate that does reach a person is explained in the ledger

What it costs

• Partitions and paging-domain outages can produce duplicates within 60 seconds
• Dedup keys depend on a stable definition of step, which constrains policy editing mid-incident
• A same-channel retry on another route after a hard failure must not be deduplicated, so the claim is released on confirmed failure

Choose differently when. If duplicate notifications carried a real cost, such as per-message charges large enough to matter at volume or a regulatory limit on repeated contact, stronger sender-side controls would be justified.

Why it holds up over time. The principle and the key definition are independent of channel and carrier. New channels get a key and a window; the rule that uncertainty resolves toward sending does not change.

Lesson. Put the idempotency check next to the side effect that matters, and decide in advance which way it fails.

ADR-20 · Aggregate carriers, own the last software hop: Asterisk, Jasmin, a push relay and a responder app

Status: Accepted · Shown on views: 04, 08, 19

Is the platform a notification provider, or an aggregator of providers?

Context. Owning delivery end to end is impossible on-premises: phones are reached through carriers, and iOS and Android devices are woken only through Apple and Google. Aggregating third-party notification services is cheap and makes provider diversity easy, but receipts are inconsistent, acknowledgement arrives through someone else's webhook, and do-not-disturb bypass is out of the platform's hands. The requirement needs acknowledgement from every channel paged on, SEV1 DND bypass, and sub-15-second p95 delivery.

Decision. Contract two carriers directly for SIP trunking and SMPP, and run the software that speaks to them: Asterisk for voice with an IVR that reads the summary and takes a keypress, Jasmin for SMS including inbound replies. Run a push relay that talks to APNs and FCM directly. Ship a thin responder app of the platform's own that receives critical alerts, shows the summary and acknowledges with one button. Use Postfix through two smarthosts for email.

How it works on-premise. Asterisk with PJSIP trunks to carrier A from cells A and B and to carrier B from cell C, using ARI to hand DTMF to the ack receiver. Summaries are rendered to audio ahead of time with eSpeak NG when the service catalogue changes, not at call time. Jasmin binds SMPP to both carriers with MO messages routed to the ack receiver. The push relay uses APNs token authentication and FCM HTTP v1 with credentials from OpenBao. The app requests Apple's Critical Alerts entitlement and Android's notification policy access for SEV1.

What it buys

• A keypress, an SMS reply and an app button are all acknowledgements the platform receives on its own infrastructure
• Receipt timing and failure codes are measured per route, not reported by a middleman
• Carrier diversity is a contract and a trunk, not a vendor's roadmap

What it costs

• Carrier contracts, number provisioning and, in some jurisdictions, sender and template registration are the platform team's work
• An app to maintain on two mobile platforms
• If Apple refuses the entitlement, iOS SEV1 bypass relies on voice from a number responders set as an emergency bypass contact

Choose differently when. An organisation permitted to use cloud services, with responders in regions where a reputable notification API meets the latency target, should aggregate and spend the saved effort on alert quality.

Why it holds up over time. SIP, SMPP, SMTP, APNs and FCM change slowly and are replaceable one route at a time. The acknowledgement contract, a signed token that any channel can return to the ack receiver, is independent of which of them delivered it.

Lesson. You cannot own the network to the phone. You can own everything up to it, and every acknowledgement coming back.

ADR-21 · Fail over on the same channel after a hard failure, to the next channel after silence

Status: Accepted · Shown on views: 20, 26, 34

How does the dispatcher use two independent providers within one escalation step without waiting for the step interval or duplicating pages?

Context. The requirement asks for failover between providers within a step when the primary reports failure or exceeds a latency threshold, and for provider health measured continuously rather than at page time. It also forbids the same page twice on the same channel within 60 seconds. A latency-triggered failover to the other provider on the same channel will produce exactly that duplicate whenever the first provider was merely slow.

Decision. Treat the two triggers differently. On a hard failure, a NACK, throttle, refused bind or refused call, retry the same channel on the other route immediately; the first attempt demonstrably did not leave. On silence past the channel's threshold, move to the next channel in the step's policy on the healthier route, not to the same channel on the other route. Route health, measured by loopback traffic, decides which route is tried first.

How it works on-premise. Loopback traffic from each cell: an SMS every ten minutes per route to a platform-owned SIM in the GSM bank at site C, and a call every fifteen minutes per route to a DID answered by Asterisk at another cell. Latency, receipt rate and DTMF success feed a route score in the paging domain. Thresholds start at 20 s for SMS receipts, 10 s to ringing for voice and 3 s for push acceptance, and are tuned from the proof phase measurements.

What it buys

• A failed carrier costs a page seconds, not a step
• Slow routes produce a call rather than a repeated SMS, which is also more likely to wake someone
• Route health is known before a real page needs it

What it costs

• Synthetic loopback traffic is a standing cost on both carriers
• Silence thresholds that are too tight produce more calls than needed
• Channel policy must order channels so that 'next' is always meaningful

Choose differently when. Where every route returns reliable, fast delivery receipts, a latency failover on the same channel becomes safe and simpler.

Why it holds up over time. The distinction between 'did not leave' and 'has not arrived' is true of every messaging system that will ever exist, and so is the answer: repeat what failed, change what is slow.

Lesson. Treat 'it failed' and 'it has not worked yet' as different events. They need different responses.

ADR-22 · Every channel acknowledges with a signed, single-use token; no login, no DNS dependency

Status: Accepted · Shown on views: 04, 19, 32

How does a responder stop an escalation from any channel they were paged on, with no web session and possibly no working corporate infrastructure?

Context. The requirement asks that acknowledgement work from every channel, a single keypress on a call and a reply on SMS, and not require a web session. Responders acknowledge from bed, from a car and during the identity provider's outage. An acknowledgement mechanism that needs login, VPN, corporate DNS or the console fails in exactly the scenarios that produce pages.

Decision. Every notification carries an acknowledgement token: a signed structure naming incident, responder, step and expiry, verifiable by any cell. Voice acknowledges by DTMF after the summary, SMS by replying with a short code tied to the token, push by an app button that posts the token, and email by a link or a reply. The ack receiver verifies the token, performs the state compare-and-set and marks the token used. The responder app carries the addresses of all three cells and tries them in order, so neither DNS nor a single cell is required.

How it works on-premise. Tokens are signed with Ed25519 keys held by every cell, rotated monthly with overlap. SMS short codes map to tokens in the paging domain for the lifetime of the step. Ack endpoints are exposed on each cell's public address behind HAProxy with rate limiting per source. The app pins the platform's CA and the three cell addresses in its build, updated by app release. Acknowledgements in minority mode are recorded locally and replayed.

What it buys

• Acknowledgement works throughout control-plane, identity and DNS outages
• A forwarded or intercepted token can acknowledge one step of one incident and nothing else
• Every acknowledgement records its channel, cell and token, for the review

What it costs

• Cell addresses changing requires an app release, so they are chosen to be stable
• SMS short codes are guessable if too short; they are six characters and rate limited
• Token signing keys are one more secret to rotate across three cells

Choose differently when. A platform whose responders always work from managed devices on a network that is itself outside the estate's failure domain could safely rely on the console and single sign-on for acknowledgement.

Why it holds up over time. The token format is channel-independent. A new channel only needs a way to carry six characters or a URL back to a cell; the verification and state change do not move.

Lesson. Make the action that stops the alarm need as little of the world as possible, and make it useless for anything else.

ADR-23 · Escalation exhaustion is a terminal state, a repeating broadcast and an incident against the platform

Status: Accepted · Shown on views: 25, 34

What happens when every step of an escalation policy has run and nobody has acknowledged?

Context. The requirement calls this the single worst outcome the platform can produce and asks for an explicit terminal state treated as an incident against the platform. Many tools close or leave such incidents silently, which means the organisation discovers the gap the next morning. Exhaustion is almost always a configuration failure, a wrong rota, stale contact data or a policy that ends too early, rather than bad luck.

Decision. Require every escalation policy to end in a final step; the editor refuses to save one without it, and the default final step is the owning team's management layer plus a broadcast to every responder in the owning team. Keep paging earlier targets as later steps begin. On exhaustion, set the escalation state to exhausted, repeat the broadcast every 5 minutes until acknowledged, and open a SEV2 incident against the platform on carrier B to the platform rotation.

How it works on-premise. Exhausted is a state in the paging domain, not a flag, so timers and dispatchers treat it as active. The platform incident is opened by the timer service directly through the engine's API in the same cell, and its dispatch is pinned to carrier B routes. The review template for exhaustion incidents includes the rotation, the contact methods tried and their ledger outcomes.

What it buys

• No incident can end in silence
• Exhaustion produces a review of the rota and contact data, which prevents the next one
• Management learns about exhaustion from the platform, not from a customer

What it costs

• A whole team can be woken by one person's dead phone
• Repeat broadcasts are expensive in voice minutes during a long gap
• Exhaustion incidents count against the platform's own reliability reporting

Choose differently when. None in this organisation. A platform that pages humans has no configuration in which silent exhaustion is the better outcome.

Why it holds up over time. The rule is independent of how policies are modelled or who is on them. Future policy features, such as follow-the-sun or automated backfill, must still end somewhere loud.

Lesson. Name your worst outcome, make it a state rather than an absence, and route it to the people who can prevent it.

Response and learning

How the response is coordinated around the incident record, and how a review is made the cheapest thing to finish.

ADR-24 · Coordination happens around the record: channels, roles, updates and ownership as events

Status: Accepted · Shown on views: 05, 10, 11

How are channels, roles, stakeholder updates and ownership handled so that coordination helps the response and the record at the same time?

Context. Above a severity threshold the requirement asks for a durable channel linked to the incident, roles held by one person at a time with handovers recorded, paging people in by name, role or service ownership, stakeholder updates as first-class objects with an audience and a next-update-by time, and routing derived from a service catalogue. Chat is also the tool most likely to be unavailable or overwhelmed during a large outage.

Decision. Create a Mattermost channel per qualifying incident asynchronously from the incident service, record the channel id on the incident, and retry until it exists; channel creation never blocks paging. Model roles, handovers, pages-in and stakeholder updates as incident events through the API. Stakeholder updates use templates, a declared audience and a next-update-by time that raises a reminder to the communications lead. Derive ownership, escalation policy and runbook from a service catalogue kept as YAML in Git and synchronised into the control plane.

How it works on-premise. Mattermost Team Edition on the organisation's existing deployment, reached through a chat bridge service with a bot account. Chat messages are not state; a slash command that changes state calls the API as the invoking user. Catalogue files follow the Backstage catalog-info format so the organisation's developer portal can read the same source. Stakeholder updates are emailed to their audience and published to the static status page at site C when marked external.

What it buys

• Who held which role when, and what stakeholders were told when, are in the timeline
• A chat outage delays a channel, not a page or a record
• Ownership changes are reviewed changes to one file

What it costs

• Responders must use commands or the console for state changes rather than free text
• Catalogue quality becomes a dependency for routing, and needs an owner
• Stakeholder templates need maintaining as the organisation's audiences change

Choose differently when. A small organisation where everyone is in one channel and there are no external stakeholders can let chat carry coordination and record only pages and state in the platform.

Why it holds up over time. Chat products change every few years; events about roles and updates do not. The bridge is the only component that knows which chat product exists, and the catalogue's file format is shared with tools beyond this platform.

Lesson. Let people coordinate wherever they talk, but record what they decided somewhere that does not depend on where they talk.

ADR-25 · Reviews are generated from the log, blameless by template, with action items tracked both ways

Status: Accepted · Shown on views: 07, 15, 31

How does the platform make the post-incident review the cheapest path, and make it change something?

Context. Reviews that start from a blank page happen late or not at all. Reviews that name people teach everyone to hide near misses. Action items filed into a tracker the platform cannot see die quietly, or are closed without any change. The requirement asks for an automatic draft with computed intervals, a template with no individual attribution field, action items synchronised with the issue tracker, overdue reviews escalated, and search across reviews for recurring factors.

Decision. Generate a draft on resolution for every incident at or above the mandatory threshold, with timeline, responders, linked changes and the intervals to detect, acknowledge, mitigate and resolve computed from events. Structure the template around contributing factors and system conditions from a controlled vocabulary plus free text, with no field for a person. Model action items as objects synchronised both ways with the tracker, and report items overdue or closed without a linked change. Index reviews for full-text and factor search in PostgreSQL.

How it works on-premise. The review module is a control-plane service writing review edits as events. Tracker synchronisation uses the tracker's webhooks inbound and REST API outbound, with a nightly reconciliation job that repairs drift and reports conflicts. Search uses PostgreSQL full-text search with pg_trgm for fuzzy matching, which comfortably covers a few thousand reviews a year. Overdue SEV1 and SEV2 reviews notify the owning team's management layer by email and chat, never by page.

What it buys

• The first draft costs the owner minutes, not hours
• Recurring contributing factors become countable across incidents
• Action items closed with no change are reported as such

What it costs

• The controlled vocabulary needs curation or it becomes a junk drawer
• Two-way sync with a tracker is a long-lived integration with conflict handling
• Some teams will want a free attribution field and will need the reason explained

Choose differently when. An organisation with a mature review practice in an existing tool, with action tracking that already works, should integrate the computed timeline into that tool rather than move reviews.

Why it holds up over time. The event log keeps the review's facts reproducible whatever the review tool becomes, and blamelessness is structural, so it survives changes of team, leadership and fashion.

Lesson. Make the right review the easiest one to write, and make the wrong one impossible to write in the template.

Security and identity

Signing in when the identity provider is down, integrations that are never trusted, contact data that can be erased from an immutable system, and incidents only some people may read.

ADR-26 · Keycloak brokers the corporate identity provider and holds break-glass accounts on hardware keys

Status: Accepted · Shown on views: 22, 33

How do responders sign in to the console when the organisation's identity provider is itself the outage?

Context. Federated authentication is required, and the corporate identity provider will sometimes be down, sometimes during the incident it is causing. The requirement asks for a break-glass path for the on-call console usable in that case, authorisation scoped by team and service, and every sensitive action audited.

Decision. Put Keycloak between the console and the corporate identity provider. Normal logins federate. A small set of local break-glass accounts, each bound to registered WebAuthn hardware keys, can sign in when federation fails. Break-glass sessions last four hours and carry a responder role that can acknowledge, resolve, change severity and post updates, but not edit schedules, escalation policies or suppression rules. Every break-glass login is an audit event and a notification to the security rotation.

How it works on-premise. Keycloak in the control-plane cluster with its database on the control-plane PostgreSQL. Groups from the identity provider map to team and service scopes through the service catalogue. Break-glass accounts, about a dozen across three timezones with two keys each, are reviewed quarterly and exercised in the control-plane rebuild drill. Keycloak is not used anywhere on the paging path.

What it buys

• An identity outage costs convenience, not the ability to run a response
• Break-glass cannot be used to make the damaging configuration changes
• Every use is attributable to a person and a hardware key

What it costs

• Keycloak is one more service to patch and operate
• Hardware keys must be issued, registered and replaced
• Break-glass holders are a small group whose availability must be planned

Choose differently when. If the identity provider were demonstrably independent of the estate's failure domain, a cloud identity service used only for this platform, for example, the break-glass path could shrink to a documented emergency procedure.

Why it holds up over time. OIDC and WebAuthn are open standards with long horizons. The identity provider behind Keycloak can change without the console changing, and the break-glass rule, fewer powers and more scrutiny, is independent of the broker.

Lesson. Every login path that depends on the thing that might be broken needs a narrower one that does not.

ADR-27 · Every integration is untrusted, independently authenticated and revocable without a deploy

Status: Accepted · Shown on views: 11, 16, 32

How are inbound integrations authenticated so that one compromised or broken integration affects only itself?

Context. Forty services and their monitoring, vendors and ticketing systems will integrate with the platform, owned by many teams with different security practices. A shared ingest token turns any leak into the ability to page anyone. The requirement asks for per-integration credentials, signature verification, independent revocation, rotation without deployment, and provider credentials held in a secret store.

Decision. Give every integration its own identity, a signing secret for HMAC over the body and a timestamp, and an allowed set of services it may raise alerts for. Hold up to two valid secrets per integration so rotation overlaps. Store secrets in OpenBao and publish only what the gateway needs to verify into the paging domain. Revocation is a control-plane action that takes effect within the coverage staleness bound. Carrier, APNs, FCM and token-signing credentials are also held in OpenBao and loaded into memory at start.

How it works on-premise. OpenBao key-value secrets for integration secrets and transit keys for wrapping the verification material published to the paging domain. Gateways verify with a 5-minute replay window. Integrations that can present client certificates may use mTLS from a per-integration certificate instead of HMAC. Email integrations authenticate by address and sender allowlist and are limited in the severity they may propose.

What it buys

• A compromised integration can raise alerts only for its own services, and is revoked alone
• Secret rotation is routine because two secrets overlap
• Ingest never waits for the identity provider or OpenBao

What it costs

• Integration owners must implement signing, which some vendors cannot
• A signing proxy is needed for sources that can only send unsigned webhooks
• Two secret stores in effect: OpenBao as source, the paging domain as the published copy

Choose differently when. A platform receiving alerts only from a single monitoring stack operated by the platform team itself could authenticate that one source with mTLS and skip per-integration management.

Why it holds up over time. HMAC signatures and mTLS will outlive every monitoring product sending them. The scope-per-integration rule holds however integrations multiply.

Lesson. Give every sender its own key and its own blast radius, and make revoking one as dull as rotating it.

ADR-28 · Contact data is encrypted per responder, so offboarding erases it from an immutable system

Status: Accepted · Shown on views: 15, 32

How can personal contact data be restricted, encrypted and erased when the platform keeps an immutable log for seven years and replicates coverage into three sites?

Context. Phone numbers and device tokens are personal data with restricted access and a required erasure path on offboarding. The same data must be in the paging path to be useful, replicated to three sites and cached in coverage buckets. The event log and the ledger are immutable by design, and ledger records name the address a page was sent to.

Decision. Encrypt every contact address with a data key specific to its responder, wrapped by OpenBao's transit engine. Coverage buckets carry method ids and the wrapped key reference, not clear addresses. The event log and ledger store responder ids and method ids only. The dispatcher unwraps and decrypts at the moment of sending. Offboarding deletes the responder's data key, after which every stored copy of their addresses, in any store and any backup, is unreadable.

How it works on-premise. Addresses encrypted with AES-GCM under a per-responder key. Wrapped keys stored beside the contact method in PostgreSQL and replicated into the paging domain. Dispatchers cache unwrapped keys in memory for the duration of a step. Key deletion is an audited control-plane action executed by OpenBao and propagated to caches within the staleness bound. Access to clear contact data in the console is limited to the responder and rotation owners, and every view is an audit event.

What it buys

• Offboarding erasure is one key deletion, verifiable and auditable
• Replicas and backups of the paging state leak nothing without the keys
• The immutable record stays immutable

What it costs

• Losing OpenBao's transit keys would make every address unreadable, so they are backed up and drilled like the event log
• Decryption at send time adds a small cost to each dispatch
• Historical reviews cannot show the actual number used for an offboarded responder

Choose differently when. If contact data were not considered personal data in the jurisdiction, or retention obligations required keeping addresses for the audit period, simpler at-rest encryption with ordinary access control would do.

Why it holds up over time. Crypto-shredding decouples erasure from every storage decision the platform will make later. Any future store that holds contact data inherits erasure for free as long as it holds ciphertext.

Lesson. When a system must never delete and must sometimes forget, encrypt per subject and delete the key.

ADR-29 · Incidents are readable by default; sensitive ones are restricted by row-level security, and the restriction is audited

Status: Accepted · Shown on views: 15, 32

How are security and personnel-adjacent incidents kept to a named group without making every other incident harder to reach?

Context. Responders need broad read access to timelines to learn and to help. A small share of incidents, security investigations and anything involving individuals, must be visible only to a named group. Restriction can also be misused to hide an embarrassing incident, so the act of restricting must itself be visible. The requirement asks for readable-by-default timelines, restriction to a named group, and the restriction audited.

Decision. Default every incident to readable by all responders. Allow the incident commander or the security rotation to mark an incident restricted to a named group, as an event with a reason. Enforce restriction in the database with row-level security on the event log and projections, not only in the console. Redact restricted incidents in the outbound stream to id, state and severity. The existence of a restricted incident is visible; its content is not.

How it works on-premise. PostgreSQL row-level security policies keyed on the incident's restriction group and the session's groups, set by the incident service from the Keycloak token. ClickHouse reports exclude restricted incident content. Restriction and unrestriction events are included in a weekly report to the security and reliability leads.

What it buys

• Restriction holds for every client of the database, including future ones
• Nobody can make an incident disappear, only restrict who reads it
• Security incidents use the same paging and review machinery as everything else

What it costs

• Row-level security adds a policy check to every query on the log
• Restricted incidents are excluded from cross-incident factor search for most users
• Groups must be kept accurate in the identity provider

Choose differently when. An organisation whose security incidents are handled by a separate team with its own tooling and legal obligations may be required to keep them out of this platform entirely.

Why it holds up over time. Enforcement in the database survives changes of console, API and reporting tool. The audited restriction event is a governance control that does not depend on any product.

Lesson. Put access control where every reader has to pass through it, and make the decision to restrict as visible as the thing restricted.

Operations and cost

How the platform proves it works from outside itself, how the paging path changes without becoming the outage, and how money and sleep are kept visible.

ADR-30 · A synthetic page runs every minute, an outside watchdog pages on carrier B, and a manual fallback kit is drilled

Status: Accepted · Shown on views: 28, 34

How does the platform prove it works, and how is its own failure detected by something that does not share it?

Context. A paging platform that alerts about itself through itself fails silently in the one case that matters. The requirement asks for a continuous end-to-end synthetic page through the real path, its failure paged through the alternate provider, and a manually operated fallback paging path for total primary failure, exercised on a schedule.

Decision. Inject a synthetic alert into every cell every 60 seconds; it becomes a test incident, pages a platform-owned SIM and a test app, and is acknowledged by a robot. Run a dead-man switch at site C that expects heartbeats from every cell and the synthetic loop, and on silence pages the platform rotation through carrier B directly, bypassing the paging services. Keep a manual fallback kit, a hardened laptop with a CLI, an hourly encrypted export of coverage and policies, and the GSM modem bank, and use it once a year to page a volunteer rotation.

How it works on-premise. The synthetic injector and robot run on hosts outside the cells at each site. The dead-man switch is a small independent service at site C with its own Asterisk dialplan and Jasmin route, sharing no process with cell C's paging services. The fallback CLI drives Gammu against the modem bank and reads the exported snapshot decrypted with keys held by two platform leads. Results of every drill are published as incidents with reviews.

What it buys

• End-to-end failure is known within about two minutes, from outside the failing system
• The fallback kit's steps, keys and data are known to work as of the last drill
• The synthetic time series is a continuous measure of the p95 target

What it costs

• Standing carrier traffic for synthetics and loopback on both routes
• Drill time from the platform team and a volunteer rotation each year
• The watchdog is one more independent component to keep independent

Choose differently when. A smaller platform with a single site could reasonably rely on an external uptime service calling a phone, if a cloud service were permitted for the watchdog alone.

Why it holds up over time. 'Test through the real path, watch from outside, exercise the fallback' holds for any future architecture. The synthetic contract, an alert in, an acknowledgement back, is the definition of the product.

Lesson. The only proof that a paging system pages is a page. Send one every minute, and have someone else watch for it.

ADR-31 · The paging path changes one cell at a time, never during a SEV1 or SEV2, from artefacts already on the host

Status: Accepted · Shown on views: 27, 09

How does the paging path change often enough to stay current without a release becoming the outage?

Context. Changes are the most common cause of outages, and three identical cells upgraded together are one cell. The platform's outage is everyone's outage, and a deploy that depends on the registry, Git or DNS cannot run during the incidents that most need a fix. The paging path still needs regular releases for security patches, tzdata and features.

Decision. Release the paging path through gated waves: staging cells, then cell C, then cell B the next day, then cell A the day after. Each wave requires three synthetic pages acknowledged through the new cell. No wave starts while a SEV1 or SEV2 is open, and none starts unless both platform-team regions are in working hours. Artefacts, configuration and tzdata are staged on every host before the switch, and rollback is a symlink and restart. The control plane deploys continuously with ordinary canaries.

How it works on-premise. Forgejo with signed commits and two reviewers, Forgejo Actions building static binaries, SBOMs and cosign signatures pushed to Harbor. Gates run schedule resolution to 2030 against both resolvers and replay a recorded storm with an injected SEV2. Ansible stages and switches host by host. The freeze check queries the paging domain for open SEV1 and SEV2 incidents directly, so it works when the control plane does not.

What it buys

• Most defects are found by a synthetic page in the outpost cell before an estate cell is touched
• Deploys and rollbacks work during infrastructure incidents
• Releases are routine enough that nobody fears them

What it costs

• A full paging-path release takes three working days
• Urgent security fixes need an expedited path, which still keeps one-cell-at-a-time
• Freezes during long incidents delay fixes that might help

Choose differently when. If the paging path grew to many cells in many locations, waves by percentage of cells would replace waves by day.

Why it holds up over time. The rules, one failure domain at a time, prove with a real page, never during a major incident, never depend on the network to roll back, apply to any deployment technology the team adopts later.

Lesson. The safest release process is the one that assumes the release is wrong and the network is down.

ADR-32 · Money and sleep are metered per service and rotation; the independence premium is a stated number

Status: Accepted · Shown on views: 26, 29

How are notification spend, standby cost and human interrupt load kept visible and attributed?

Context. Voice and SMS are the platform's dominant marginal cost and are driven by alert quality, which the platform team does not control. The outpost is insurance whose premium must be a known number. Out-of-hours pages and non-actionable pages are the cost to people, and the requirement asks for them to be measured and reported, not enforced.

Decision. Write a cost on every ledger attempt from the carrier rate card at send time. Attribute spend, pages, out-of-hours interrupts and actionable rate to service, rotation and responder. Deliver one weekly report per service owner combining noise, cost and interrupts. State the outpost's yearly premium in the platform budget as insurance, currently estimated at about USD 55,000 before staff time, and review it yearly against the incidents it covered. Prefer push and email over voice where severity policy allows, and never weaken SEV1.

How it works on-premise. Rate cards are versioned control-plane data. ClickHouse materialised views compute weekly aggregates; a job renders the report and emails it, and Grafana dashboards show the same data. Actionable pages are those followed within the incident by a linked change, a severity change, a note or a recorded decision. Synthetic and loopback traffic are attributed to the platform's own budget.

What it buys

• Service owners see what their alerts cost in money and in their colleagues' nights
• The outpost survives budget reviews because its cost and value are both visible
• Severity channel policy can be tuned against measured spend

What it costs

• Rate cards must be kept current with carrier contracts
• Actionable is a heuristic and will be argued with
• Reporting without enforcement depends on management attention

Choose differently when. If carrier traffic were effectively free under a flat contract, spend attribution would matter little and the report could focus entirely on interrupts.

Why it holds up over time. Cost per attempt and interrupts per responder are measures that stay meaningful whatever the carriers, channels or organisation structure become. Stated premiums are how insurance survives changes of leadership.

Lesson. Put a price on noise where the people making it can see it, and put a price on insurance before anyone asks what it is for.

Every package used, in one table

Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.