Incident Management Platform  ·  View 27 of 34  ·  6 · Operations

Changing the Paging Path — One Cell at a Time, Never During a SEV1

The route a change to the paging path takes from commit to the third cell, and the gates that decide whether it moves on.

Editable source SVG draw.io All views
Source Forgejo signed commits · 2 reviewers Forgejo Actions static Go binaries Prove Schedule tests every DST to 2030 Storm replay recorded 20k/s burst Sign cosign · SBOM Harbor OCI artefact Staging cells synthetic pages Gate Change freeze check no open SEV1 or SEV2 Business hours both regions awake Cell C Ansible · cell C host by host Synthetic gate 3 pages acked Cells B then A Ansible · cell B next day Ansible · cell A day after fail Changing the Paging Path — One Cell at a Time, Never During a SEV1 Data store Application we own Decision point Security / platform failure / alternate Binaries and configuration are staged on every host before a switch, so a deploy needs no registry, DNS or Git reachable at the moment it happens. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • The outpost cell is changed first, because it is the cell whose failure is best covered by the other two during normal operation. Cell B follows the next day and cell A the day after (ADR-31).
  • No paging-path change starts while a SEV1 or SEV2 is open, and none starts outside hours when both the platform team's regions are awake. A deploy is the most common cause of an outage, and this platform's outage is everyone's.
  • Binaries, configuration and tzdata are staged on every host before the switch. A deploy needs no registry, Git or DNS reachable at the moment it happens, and rollback is a symlink and a restart.

Gates

  • Schedule resolution for every rotation across every DST transition to 2030. Replay of a recorded storm with an injected SEV2 dispatched on time. Three synthetic pages acknowledged through the new cell before the next cell is touched.

Control plane

  • Control-plane services deploy continuously through Argo CD with ordinary canaries. Their failure costs the console for minutes, which the degradation order allows.
◀  Deployment — Two Estate Sites and an Outpost That Shares Neither All views Observability — Watching the Watcher From Outside  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises27-release-pipeline