Incident Management Platform  ·  View 10 of 34  ·  3 · Structure

Container Architecture — The Control Plane

The services that edit schedules, run the response, hold the record and write reviews. It runs on a small Kubernetes cluster the platform team owns, at one site, with a warm rebuild at the other.

Editable source SVG draw.io All views
Control plane · site A · platform-owned Kubernetes, warm rebuild at site B Response Incident service state · roles · updates Console TypeScript web UI Review module drafts · actions Scheduling Service catalogue YAML in Git Schedule resolver 72 h · 14 d gaps DST verifier second implementation Data PostgreSQL CloudNativePG · sync Projector rebuildable views ClickHouse ledger · noise · cost Integration and identity Keycloak broker · break-glass Chat bridge channel per incident Tracker sync two-way · reconciled Paging cells coverage KV · outbox Corporate IdP OIDC · SAML publish snapshot append federate Container Architecture — The Control Plane Application we own Decision point Data store Security / platform Interface / broker External / third party event / async synchronous Omitted for legibility: the reconciler reading the paging outbox (view 18), the chat bridge and tracker sync reaching Mattermost and the tracker (view 11), and OpenBao, which every service reads at start. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Single-site with a warm rebuild, not active-active. The requirement gives the control plane a 15-minute RTO, and a hot second control plane buys nothing the requirement asks for while doubling what has to be kept consistent (ADR-03).
  • A platform-owned cluster rather than the shared estate clusters. It costs one more small cluster to run; it means a Kubernetes incident in the estate cannot take the console with it at the moment responders need it.
  • The DST verifier is a second, independent implementation of schedule resolution. Two resolvers that agree are much harder to fool than one resolver with good tests (ADR-15).

Targets

  • Control plane ≥ 99.9% monthly, RTO ≤ 15 minutes. Console incident list p95 ≤ 1,500 ms. Event log RPO ≤ 5 s, met in practice at 0 by synchronous replication to site B.

Risks

  • A warm rebuild that is never exercised is a hope. The site-B rebuild runs quarterly from the manifests and the standby, timed, with the result published.
◀  Container Architecture — One Paging Cell All views Integration Surface — Every Interface  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises10-control-plane