Backup and Restore Service  ·  View 18 of 26  ·  5 · Runtime

What Keeps Working When Parts of the Platform Are Gone

Four partial outages, and for each one exactly what continues, what is blocked, and what pages.

Editable source SVG draw.io All views
Control plane down Catalogue lost Data centre A lost OpenBao unreachable Tier 1 log shipping Continues engine-owned Continues Ships to B repo Continues key loaded at start Tier 1–2 base copies Fail-static CronJobs Continue · events queue Run against B Blocked · RPO at risk Tier 3 sweeps Paused first Continue Paused Blocked Restore Manual runbook S3 + tool + key Envelope scan Orchestrator in B Offline key recovery Verification Stops · heartbeat pages Runs · proofs buffered Runs from B Stops · heartbeat pages Reporting Stale · banner Rebuilt last From B replica Unaffected What Keeps Working When Parts of the Platform Are Gone Degrade order: reporting, console, Tier 3 capture, verification. Tier 1 capture and restore never. v 1.0 · owner SRE · date 2026-09

Decisions

  • Fail-static base copies: the policy compiler renders every Tier 1 and 2 schedule into suspended Kubernetes CronJobs beside the agents. A watchdog unsuspends them if Temporal's heartbeat is missing for two intervals. They are removed when Temporal returns, and their captures are replayed into the catalogue from their envelopes.
  • Degraded mode is always announced. Every affected datastore's status shows a banner naming the missing component, because a stale green status is worse than an honest amber one.
  • Losing OpenBao is the one outage that blocks capture and restore together. That is why it has 5 voters across three sites and its own quarterly recovery drill.

Order of degradation

  • Reporting first, then the console, then Tier 3 capture, then verification. Tier 1 capture and the restore path never degrade by design. If they fail, it is an incident, not a mode.

Risks

  • Adapters load the key at start, so a long OpenBao outage becomes a problem when agents restart, not immediately. Agent restarts are frozen during an OpenBao incident, and the runbook says so.
◀  In-Place Restore — Authorised, Snapshotted, Reversible All views Deployment — Two Data Centres and a Vault Site  ▶
Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises18-degraded-modes