# Architecture Decision Record *Backup and Restore Service · Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises* The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager). Twenty-six decisions make up this architecture. Everything else on the twenty-six views is convention, and convention needs no defence. Each record starts with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then gives the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the tools, the hardware and the team have all changed. Read the one-pager first. Read a record when you want to argue with it. > **Status of this document.** This is a design, not a report on a running system. Throughput, capacity and duration figures are sized targets and stated assumptions taken from the requirement, and each one is replaced by a measured value at the first drill. Four product behaviours are relied on and must be confirmed in the proof of concept before any production copy depends on them: Ceph RGW honouring governance bypass only for a caller with the explicit permission, RGW STS web-identity roles accepting SPIRE-issued JWT-SVIDs, ClickHouse BACKUP to an encrypted disk over S3, and TiDB BR log backup writing to a bucket with Object Lock enabled. ## How to read a record - **Question:** The forcing question: why a decision was needed at all. - **Context:** The requirement, the scale and the constraint that make it hard. - **Decision:** What this architecture does, stated so it can be checked. - **How it works on-premise:** The concrete mechanism: which package, configured how, on whose hardware. - **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each. - **Consequences:** What the choice buys and what it costs, both kept visible. - **Choose differently when:** The conditions that would flip the decision for your system. - **Why it holds up over time:** What keeps the decision right as scale, staff and technology change. - **Lesson:** The principle that transfers beyond this platform. ## Decision map **The proof**: What makes a datastore count as recoverable, how deep that proof goes, and who notices when the proving stops. - ADR-01 · Protection status is written by the verifier, never by a capture job - ADR-02 · Verification depth is chosen per tier by the failure class it catches - ADR-03 · Rehearsals use the emergency restore path, to a random second, alternating custody copies - ADR-04 · Page on the verifier's silence, from a monitor outside the platform - ADR-05 · Rehearse in a dedicated enclave with no route to production or the internet **Custody**: Where the copies live, who can and cannot destroy them, and the second custodian that fails differently. - ADR-06 · Custody on Ceph RGW with Object Lock, in a separately administered domain - ADR-07 · Governance-mode locks, reduced only through a sealed OpenBao needing two of five officers - ADR-08 · Two independent custody clusters joined by a verifying copier, not RGW multisite - ADR-09 · WORM tape at a third site as the second custodian, with its own rehearsed restore **Capture**: How six kinds of engine are copied consistently, at the right version, without the platform reimplementing any of them. - ADR-10 · Engine-native tools behind one adapter contract and one envelope - ADR-11 · Point-in-time recovery is each engine's capability, orchestrated rather than reimplemented - ADR-12 · Refuse to capture with a tool and engine version pair that has not been certified by a restore - ADR-14 · Capture buckets from change notifications, reconciled by a weekly listing - ADR-25 · Consistency groups by marker transactions, only for applications that cannot tolerate skew **Records**: What the catalogue is, what it is not, and the evidence that must outlive the copies. - ADR-15 · Artefacts are authoritative and self-describing; the catalogue is a rebuildable index - ADR-16 · The catalogue is an append-only event log in PostgreSQL with projections - ADR-17 · Evidence is written before action, kept 7 years under a compliance lock, apart from the catalogue **Restore**: How a copy becomes a running system inside its RTO, including the dangerous and the unknown cases. - ADR-18 · Side by side by default; in-place needs a second approver and a mandatory snapshot - ADR-19 · RTO is a budget, and hourly block-incremental bases keep log replay inside it - ADR-20 · Find the last clean recovery point by parallel nine-way search against an owner invariant - ADR-21 · A manual restore path that needs only custody, the envelope, the key and the engine tool **Control and operations**: What orchestrates the work, how protection is declared, and where everything runs. - ADR-22 · Temporal for durable workflows, with fail-static CronJobs for base copies - ADR-23 · Protection as policy on a declared tier, with no default and continuous discovery - ADR-26 · Two data centres and a vault site, a warm control plane, and a dedicated backup network **Identity and keys**: Who may do what to which copy, and how cryptography enforces the separations that policy alone would not. - ADR-13 · Encrypt at the source with data keys wrapped by per-classification keys in OpenBao - ADR-24 · Four disjoint custody roles assumed with SPIRE workload identity; no long-lived credentials ## Technology by capability Every capability on the views, the package that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns, with no managed service and no external network dependency. | Capability | Choice | Origin | Credible alternative | Why this one | Record | |---|---|---|---|---|---| | Custody object storage | Ceph RGW (Squid line), S3 Object Lock, EC 8+3 | Open source | MinIO; Garage; SeaweedFS | Mature Object Lock and STS behind the S3 API, a very large production base, and governance that has not changed hands | ADR-06 | | Two-person lock reduction | Dedicated sealed OpenBao, Shamir 2 of 5 | Open source | Custom approval service holding a bypass key; HSM quorum cards | Uses an audited, existing threshold-cryptography primitive instead of inventing one | ADR-07 | | Custody replication | Custody copier (Go), verifying SHA-256 | This design | RGW multisite zone sync; rclone | Keeps the two clusters in separate administrative realms and checks every byte before it counts as a second copy | ADR-08 | | Second custodian | Bareos with LTO-9 WORM media | Open source | Third-site Ceph cluster; Bacula | Different media, different software, offline once vaulted | ADR-09 | | PostgreSQL capture and PITR | pgBackRest 2.5x, block incremental, archive-async | Open source | WAL-G; Barman | Parallel restore, block incremental, repository encryption and a built-in verify command | ADR-10 | | MySQL capture and PITR | Percona XtraBackup 8.4 plus binlog streaming | Open source | MySQL Shell dump; mydumper | Physical hot backup with fast restore and binlog position recorded for replay | ADR-11 | | Distributed SQL capture and PITR | TiDB BR snapshot plus log backup | Open source | Logical Dumpling export | Consistent at a cluster-wide timestamp, with native point-in-time restore | ADR-11 | | Analytics capture | ClickHouse BACKUP with base_backup to an encrypted S3 disk | Open source | clickhouse-backup (Altinity) | Engine-native, part-level incremental, no extra daemon | ADR-10 | | Volume and config capture | Velero with Kopia data mover, CSI snapshots | Open source | Kasten K10 (commercial); Stash | Kubernetes-native, deduplicating, encrypted repository, freeze hooks | ADR-10 | | Bucket capture | Object mirror (Go) consuming RGW notifications via Kafka | This design | Nightly rclone listing sync | Work in proportion to what changed, not to how many objects exist | ADR-14 | | Version certification | Restore matrix in GitLab CI; signed pair list in Git | This design | Pin versions by convention | Turns silent drift into a blocked capture on the same day | ADR-12 | | Workflow orchestration | Temporal on PostgreSQL | Open source | Argo Workflows; Kubernetes CronJobs alone | Durable timers, signals for approvals and retries that survive worker restarts | ADR-22 | | Catalogue | PostgreSQL 17 with Patroni, append-only events | Open source | Kafka with ksqlDB; ClickHouse | 30 M rows is an ordinary index workload; transactional projections are simple | ADR-16 | | Evidence store | Compliance-mode locked bucket, hash-chained daily segments | This design | Audit table in the catalogue only | Evidence must not share the catalogue's failure or its administrators | ADR-17 | | Key management | OpenBao Transit, 5 voters over three sites | Open source | HashiCorp Vault (BSL); HSM with PKCS#11 | Envelope encryption, key versioning and rewrapping, community governed | ADR-13 | | Workload identity | SPIRE JWT-SVIDs to RGW STS roles | Open source | Static RGW keys in Kubernetes secrets | No long-lived custody credential exists to steal | ADR-24 | | Human identity and approvals | Keycloak with WebAuthn | Open source | LDAP with OTP | Phishing-resistant second person for in-place restore approvals | ADR-18 | | Restore authorisation | Open Policy Agent | Open source | Rules coded in the API | Owner and classification rules as reviewable, testable policy | ADR-18 | | Dead-man monitor | Healthchecks (self-hosted) on the shared observability estate | Open source | Prometheus absent() rules | Push-based absence detection, hosted by another team | ADR-04 | | Isolated rehearsal | Dedicated Kubernetes cluster, Cilium deny-all egress, own VRF | Open source | Namespace in a shared cluster | A full copy of production needs a network boundary, not a label | ADR-05 | | Policy and inventory | Git plus Argo CD, NetBox, Kyverno admission | Open source | Per-datastore job configuration in a UI | Protection bound to a class and tier, reviewed like code, inherited by new resources | ADR-23 | | Observability | Prometheus, Grafana, Loki, Alertmanager (shared) | Open source | A dedicated stack for the platform | Existing on-call routes; only the dead-man check needs independence | ADR-04 | ## The decisions, and the alternatives that lost ### The proof *What makes a datastore count as recoverable, how deep that proof goes, and who notices when the proving stops.* #### ADR-01 · Protection status is written by the verifier, never by a capture job **Status:** Accepted · **Shown on views:** 02, 11, 22 *What event makes the platform say a datastore is protected?* **Context.** Capture jobs report success when bytes are written. A copy can be written cleanly and still be unrestorable: the wrong engine version, a truncated chain, a lost key, or a restore far slower than the RTO. Most of the loss this platform exists to prevent looked fully protected on a job-status dashboard. **Decision.** A datastore is PROVEN only when a rehearsal restored it at its declared depth inside the verification interval. Proof expires at twice that interval. Capture success moves a datastore to Captured, which is displayed as unproven. last_backup_succeeded does not appear in the status projection at all. **How it works on-premise.** The verifier appends a verified event carrying depth, assertion counts, measured duration and custody location to catalogue_event in PostgreSQL. protection_status is a projection maintained in the same transaction. A Prometheus SQL exporter publishes the age of last_proven_restore, and Alertmanager pages the owner when it passes twice the interval. | Option | Verdict | Reasoning | |---|---|---| | Verifier-written status with expiring proof | Chosen | The only option where the thing reported is the thing that matters on the day of need. | | Job success as status, rehearsal as a separate report | Rejected | Two reports, and in practice everyone reads the green one. | | Job success plus periodic checksum verification | Rejected | Proves bytes are intact, not that a system comes back. It misses version drift, key loss and slow restores. | | Status from job success for low tiers only | Right elsewhere | Reasonable for scratch data with no RTO. Here, Tier 3 still gets sampled proof, because it is 1.3 PB of someone's data. | **What it buys** - The dashboard and the day of need agree - Rehearsal failures surface as normal, owned work - Auditors get a date and a measured duration **What it costs** - A rehearsal pipeline must exist before anything is called protected - Status looks worse than job-success dashboards at launch, which needs managing - Verification becomes the largest recurring cost **Choose differently when.** Choose job-based status only for data whose loss the business has explicitly accepted, such as caches and reproducible build outputs, and label that status 'captured' rather than 'protected'. **Why it holds up over time.** It does not matter which engine, tool or storage medium is in use: a copy is only as good as the last restore from it. Tools will change several times over the life of this platform, and this rule will not need to. > **Lesson.** Report the outcome you are promising, not the step that usually comes before it. #### ADR-02 · Verification depth is chosen per tier by the failure class it catches **Status:** Accepted · **Shown on views:** 15, 22 *What does 'verified' actually mean, and should it mean the same thing for every datastore?* **Context.** Each extra depth of assertion catches a different failure and costs roughly an order of magnitude more. A restore that completes catches unreadable copies. Structure checks catch partial restores. Content checks catch silent loss. Booting the application catches copies that are intact but unusable. Running depth 4 for 120 datastores daily would need a second production estate. **Decision.** Tier 1 is rehearsed daily at depth 4, Tier 2 on a rotation at depth 3 with every datastore proven within 30 days, and Tier 3 by 0.1% daily sampled read-back at depth 2 plus one full bucket restore per quarter at depth 3. The depth actually reached is recorded on every proof. Lowering depth or frequency requires the tier's RPO and RTO owner to accept it in writing. **How it works on-premise.** The assertion runner is a container that runs engine-specific checks. Depth 2 checks the schema migration version, object counts and partition coverage. Depth 3 runs owner-supplied invariant SQL, counts within tolerance, and a checksum over sampled cold key ranges compared to a probe taken on the source at a recorded log position. Depth 4 deploys the owner's Helm chart into the enclave against the restored copy and runs their smoke-suite container. | Option | Verdict | Reasoning | |---|---|---| | Depth per tier, depth recorded per proof | Chosen | Spends verification money where the RTO is short and the data is critical. | | Depth 1 everywhere, daily | Rejected | Cheap, and blind to most of the failures that matter. | | Depth 4 everywhere, weekly | Rejected | Unaffordable at this estate size, and the gap is too long for Tier 1. | | Statistical sampling across all tiers | Right elsewhere | Right for very large fleets of identical datastores. This estate has few, dissimilar ones. | **What it buys** - Cost is proportional to criticality and argued in one place - Each depth maps to named failures, so a gap is visible - Owners can see what a cheaper tier gives up **What it costs** - Depth 3 and 4 need owners to write invariants and smoke suites - Tier 3 proof is probabilistic and has to be explained with its confidence figure - Depth 4 needs a deployable application in an isolated network **Choose differently when.** Raise Tier 2 to depth 4 if a real incident shows an intact-but-unusable copy there. Drop Tier 1 to depth 3 for a datastore whose application genuinely cannot boot in isolation, with the gap written against the tier. **Why it holds up over time.** The failure classes (unreadable, incomplete, silently wrong, unusable) are properties of data and applications, not of any tool. As hardware gets cheaper, the per-tier depths can be raised without changing the structure. > **Lesson.** Decide what each check proves before deciding how often to run it. #### ADR-03 · Rehearsals use the emergency restore path, to a random second, alternating custody copies **Status:** Accepted · **Shown on views:** 14, 15, 19 *Should verification run through its own optimised path, or through exactly the path a real restore would use?* **Context.** It is tempting to verify by reading a snapshot directly or mounting a volume, which is fast and cheap. But a verification path that differs from the emergency path proves only the verification path. Restores to 'now' never exercise long replay, and restores from the nearest custody cluster never measure the cross-site one. **Decision.** Rehearsals start the same Temporal RestoreWorkflow as an emergency, with the same adapters, identities, key unwrap and custody reads. Only the target (the enclave) and a purpose field differ. Tier 1 targets a random second in the last 24 hours. Rehearsals alternate between custody A and custody B. Durations count toward published RTO attainment. **How it works on-premise.** The verifier schedules Temporal workflows with purpose=drill and target=enclave. The restore activities are the same code as for emergencies, and a CI rule fails any change that adds a purpose branch outside target selection. Custody location is recorded on the proof, and the enclave in DC-B reads A over the inter-site link. | Option | Verdict | Reasoning | |---|---|---| | Same path, random point, alternating copy | Chosen | Every drill is a real restore, so its duration is a real RTO sample. | | Dedicated fast verification path | Rejected | Proves the fast path works, which nobody will use in an incident. | | Restore to latest point only | Rejected | Never exercises the replay that old recovery points need. | | Verify in production against a live replica | Rejected | Puts a copy of production and restore traffic inside the production network, and proves nothing about custody. | **What it buys** - Measured RTO is a byproduct of verification - Regressions in the restore path show up the next morning - Cross-site restore throughput is known, not assumed **What it costs** - Rehearsals are as slow as real restores - Custody reads across sites cost link capacity every other day - Restore path changes need drill capacity to test **Choose differently when.** A separate path is reasonable where the restore procedure is purely manual and cannot be automated, but then the drill is a timed human exercise, and it should be run more often. **Why it holds up over time.** However the restore path changes, the drill follows it automatically because it is the same code. Nothing needs to be kept in step by hand, and code that has to be kept in step by hand eventually drifts. > **Lesson.** Test the path you will actually use, or you have only tested your test. #### ADR-04 · Page on the verifier's silence, from a monitor outside the platform **Status:** Accepted · **Shown on views:** 15, 21, 01 *How do we find out that verification has quietly stopped?* **Context.** The most dangerous failure of a verification pipeline is not a failed rehearsal, which pages. It is the pipeline not running: a suspended schedule, an expired credential, or a notification route dropped upstream. To a monitor that watches only for failures, all of these look like health. A monitor inside the platform fails along with it. **Decision.** Every successful rehearsal pings a Healthchecks check, one per tier and one per Tier 1 datastore, with a grace period of twice the interval. Healthchecks is operated by the observability team on the shared estate and pages through their routes. Silence is a page, not a ticket. **How it works on-premise.** A self-hosted Healthchecks instance (BSD licence) runs on the observability team's cluster in DC-B, backed by their PostgreSQL. The verifier calls it with a plain HTTPS GET to the check's URL after writing the proof, with no platform identity involved. A Prometheus blackbox probe from the same team monitors Healthchecks itself. | Option | Verdict | Reasoning | |---|---|---| | Push-based dead-man check operated by another team | Chosen | Independent failure domain, and alerts on absence by design. | | Prometheus absent() or timestamp-age rules in our own stack | Rejected | Fails with the platform's own scrape path and is easy to silence with the platform's own access. | | Daily email report of rehearsals run | Rejected | Nobody notices the email that did not arrive. | | A commercial SaaS dead-man service | Right elsewhere | A fine choice where outbound internet from the monitoring estate is acceptable. It is excluded by the on-premises constraint here. | **What it buys** - A stopped pipeline pages within two intervals - The alarm does not share credentials or infrastructure with the thing it watches - One more organisation has visibility of recoverability **What it costs** - A cross-team dependency with its own change process - Check lists must be kept in sync as datastores are enrolled - A monitor to be monitored **Choose differently when.** If there is no independent observability team, host the check at the vault site on separate hardware, run by custody officers. What matters is that it is independent, not which team owns it. **Why it holds up over time.** Alerting on absence is a pattern that does not depend on any tool. Healthchecks can be replaced by any push-based check without the verifier changing more than a URL. > **Lesson.** The alarm for 'nothing happened' has to live somewhere that keeps running when you do not. #### ADR-05 · Rehearse in a dedicated enclave with no route to production or the internet **Status:** Accepted · **Shown on views:** 19, 24 *Where do daily restores of full production data go?* **Context.** Rehearsals restore full copies of production every day into the environment that is created most often and watched least. A namespace in a shared cluster is one bad network policy away from connecting a restored payments database to something that can reach it, or reach out from it. **Decision.** A dedicated Kubernetes cluster in its own VRF in DC-B, with deny-all egress at both the network and the Cilium layer, no production credentials, and a single read-only proxy to custody. Restored data is destroyed at the end of each rehearsal, and access to a restored copy is an audited event. Depth-4 applications run with enclave-only configuration. **How it works on-premise.** 16 NVMe nodes. The data-centre firewall permits the VRF to reach only the custody read proxy and the internal registry mirror. CiliumNetworkPolicy defaults to deny with a cluster-wide rule. Kyverno rejects Secrets that reference production credential paths. Volumes are provisioned from local NVMe and wiped with blkdiscard when released. | Option | Verdict | Reasoning | |---|---|---| | Dedicated cluster in its own VRF | Chosen | A network boundary enforced below Kubernetes, so a misconfiguration inside the cluster cannot open a route out. | | Namespace in a shared non-production cluster | Rejected | A label is not a boundary for data this sensitive. | | Restore into the production cluster | Rejected | Puts restored data one DNS name away from real clients. | | Ephemeral virtual machines per rehearsal | Right elsewhere | Good isolation, but slower to start for depth-4 application boots at this cadence. | **What it buys** - Rehearsal cannot leak data or send to real downstream systems - Security review happens once, for a fixed environment - Wiping is automatic **What it costs** - A second cluster to run and patch - Depth-4 smoke suites cannot call real dependencies and need stubs - An image registry mirror inside the enclave **Choose differently when.** If masking obligations require it, run content assertions on masked copies. In that case the enclave's isolation matters less than the masking pipeline, and that pipeline must be rehearsed too. **Why it holds up over time.** The risk (full copies of production in an environment used every day) grows with every new datastore, not with any particular technology. A network boundary below the orchestrator remains valid whether rehearsal later moves to virtual machines or to something else. > **Lesson.** The environment holding copies of everything deserves the same security review as production. ### Custody *Where the copies live, who can and cannot destroy them, and the second custodian that fails differently.* #### ADR-06 · Custody on Ceph RGW with Object Lock, in a separately administered domain **Status:** Accepted · **Shown on views:** 06, 07, 19, 24 *Where do copies live so that nothing able to destroy the original can reach them?* **Context.** The copies must survive a compromised backup platform, a stolen credential and a well-meaning operator having a bad day. Immutability implemented in application code protects only against callers who use that code. The requirement puts the failure domain at the account boundary, which on-premises means a separate administrative domain: hardware, credentials and staff. **Decision.** Artefacts are written to Ceph RGW buckets with versioning and Object Lock default retention applied at write time, on racks, switches, bastions and a Keycloak realm that the backup platform and production administrators cannot access. The platform reaches custody only through the S3 API with STS roles. Custody root access is two-person. **How it works on-premise.** Custody A has 18 hosts of 24 × 22 TB HDD with erasure coding 8+3 and 8 RGW gateways behind HAProxy on the backup network. Bucket defaults set the retention mode and period per tier. Versioning means a tool rewriting a manifest creates a new version and the locked version remains. Host root access requires SSH certificates issued only against two approvals, and RADOS-level object operations outside RGW raise an alert. | Option | Verdict | Reasoning | |---|---|---| | Ceph RGW with Object Lock in a separate domain | Chosen | Storage-enforced immutability through the most common storage API, on a project with stable governance. | | MinIO with Object Lock | Rejected | Technically capable, but the community edition's scope and distribution changed during 2025, which is the wrong profile for a store meant to last a decade. | | Backup appliance with a proprietary immutable store | Rejected | Conflicts with the open-source constraint, and its restore path depends on one vendor's software. | | Deduplicating backup server on a filesystem (Bareos disk, restic REST) | Right elsewhere | Simpler for small estates. The immutability is weaker and it does not scale to 5 PB. | **What it buys** - A stolen capture credential cannot delete or overwrite a locked version - Custody can be replaced by any S3 Object Lock implementation - Erasure coding keeps the raw-to-usable ratio at 1.375 **What it costs** - Ceph needs skilled operators, and the custody team must include them - Object Lock protects against S3 callers, not against storage root - Versioned buckets grow with every manifest rewrite until expiry **Choose differently when.** For estates under about 100 TB, a pair of well-separated backup servers with append-only repositories and tape may be simpler and adequate. Above a few petabytes with a strong compliance mandate, adding a third-site cluster is the next step. **Why it holds up over time.** The S3 API with Object Lock is the most widely implemented form of object storage, available from several independent open-source and commercial implementations. Because the platform uses only that API, custody can move to another implementation without redesign. Separating administrative domains is an organisational control and does not depend on the storage technology. > **Lesson.** Put immutability where the attacker's credential does not reach, and be honest about who can still reach below it. #### ADR-07 · Governance-mode locks, reduced only through a sealed OpenBao needing two of five officers **Status:** Accepted · **Shown on views:** 13, 25 *Who can shorten a retention lock, release a legal hold, or delete before expiry, and how do we make sure that is never one person?* **Context.** Compliance-mode locks cannot be shortened by anyone, which makes lawful erasure and correcting a wrong tier impossible for up to seven years. Governance mode can be bypassed by any caller with the bypass permission, so whoever holds that permission is a single point of destruction. A workflow that asks for two approvals but then uses one service credential only moves that single point somewhere else. **Decision.** Tier retention uses governance mode. The bypass permission is held by no standing identity. It is issued by a dedicated OpenBao instance in the custody domain that stays sealed. Unsealing needs 2 of 5 Shamir shares held by custody officers, neither of whom may be the requester. Once unsealed, it issues one 15-minute credential scoped to the listed object versions, and it is sealed again straight away. The evidence store uses compliance mode. **How it works on-premise.** The break-glass OpenBao runs on two small hosts in the custody racks with integrated storage and auto-unseal disabled. The retention broker (Go) records intent with the object-list hash, collects approvals through Keycloak WebAuthn, waits for the unseal, requests the credential from a role whose policy is templated to the object list, performs PutObjectRetention with the bypass header, then calls seal. Placing a hold or extending retention needs none of this. | Option | Verdict | Reasoning | |---|---|---| | Governance mode plus threshold-sealed issuer | Chosen | Two people are needed by construction, and disposal is still possible when the law requires it. | | Compliance mode for everything | Rejected | A mistaken seven-year lock cannot be released, and erasure obligations cannot be met. | | Approval workflow holding a standing bypass key | Rejected | Whoever compromises the workflow's host gets the bypass, with or without approvals. | | HSM with an m-of-n card quorum | Right elsewhere | Equivalent strength, better physical tamper resistance, and a higher cost and procurement burden. | **What it buys** - No person or component alone can reduce retention - The mechanism is an audited open-source primitive, not new cryptography - Legal disposal remains possible **What it costs** - Every reduction takes about an hour of two officers' time - Share custody must be kept up through staff turnover - If three of five holders are unavailable, reductions stop until shares are re-issued **Choose differently when.** If the regulator requires records that cannot be altered at all and there is no erasure obligation, use compliance mode for that set and accept that a mistake lasts until expiry. **Why it holds up over time.** Threshold secret sharing is decades old and does not depend on a product. The design depends only on 'sealed until k of n agree', which any successor key manager or HSM provides. Share re-issue on role change keeps the control valid as people come and go. > **Lesson.** A two-person rule enforced by software one person administers is still a one-person rule. #### ADR-08 · Two independent custody clusters joined by a verifying copier, not RGW multisite **Status:** Accepted · **Shown on views:** 02, 12, 19 *How does a Tier 1 copy reach the second data centre?* **Context.** The requirement asks for Tier 1 in two locations. Ceph's multisite replication does this natively, but it joins both clusters into one realm with shared metadata and an administrator who can act on both. It also copies whatever arrives, including a corrupted object, without checking it. **Decision.** Custody A and custody B are separate Ceph clusters with separate realms, keys and administrators. A custody copier with a read role on A and a write-only role on B copies each artefact, recomputes SHA-256 against the envelope, and only then records a copied event. Tier 3 goes to the custody cluster in the data centre opposite its source. **How it works on-premise.** The copier is a Go service driven by captured events from the catalogue, with a lag target under 15 minutes for Tier 1. It holds two STS sessions, one per cluster, from different trust configurations. A checksum mismatch is a durable finding and the copy is retried from A. B's bucket defaults set their own Object Lock retention independently of A. | Option | Verdict | Reasoning | |---|---|---| | Independent clusters with a verifying copier | Chosen | Two administrative domains, and the second copy is checked before it counts. | | RGW multisite zone replication | Rejected | One realm and one blast radius, and nothing is verified end to end. | | Capture agents write to both clusters | Rejected | Doubles source-side load and puts write access to both custodians on every source host. | | rclone sync on a schedule | Right elsewhere | Fine for small estates, but it lists everything and has no event record per artefact. | **What it buys** - Compromising custody A's administration does not reach B - Every second copy has been checked against its checksum - Copy lag is a monitored number per tier **What it costs** - Own code to write and run - The copier's two roles need careful review - Twice the Ceph operational surface **Choose differently when.** Use native multisite where both sites really are administered by one team under one risk model and speed of replication matters more than independence. **Why it holds up over time.** Copy, verify, then count is independent of storage technology. If either cluster is replaced, the copier changes one endpoint. > **Lesson.** A second copy is independent only if a different set of people and credentials can destroy it. #### ADR-09 · WORM tape at a third site as the second custodian, with its own rehearsed restore **Status:** Accepted · **Shown on views:** 12, 13, 15, 19 *What takes the role of the off-provider copy for an estate that is already on-premises?* **Context.** In the cloud version of this requirement, the second custodian protects against losing the provider or the account. On-premises, the equivalent common failure is the storage software and its administrators: a Ceph bug, a compromised Ceph administrator, or one team's mistake across both clusters. The second custodian must not share any of those. **Decision.** Tier 1 weekly fulls and the regulatory set's monthly copies are written by Bareos to LTO-9 WORM cartridges, vaulted at a third site under custody officers' control. Once tape is live, one Tier 1 datastore is restored from tape every quarter in the enclave and timed. Writing to tape starts in Phase 3, as the requirement stages it, but the library is ordered during the MVP because tape hardware takes the longest to procure. **How it works on-premise.** A tape library with 4 LTO-9 drives at the vault site, a Bareos director and storage daemon on separate hosts, and a staging pool that reads already-encrypted artefacts from custody B with a read role. Cartridges are write-once media and leave the library on a weekly rotation. Media is migrated every two LTO generations, and a working drive of the old generation is kept until migration is proven by a restore. | Option | Verdict | Reasoning | |---|---|---| | WORM tape via Bareos, vaulted | Chosen | Different media, software and administrators, and offline once vaulted. | | Third Ceph cluster at the vault site | Rejected | Shares the software, bugs and skills of the first two. | | Commercial cloud object storage with Object Lock | Right elsewhere | A strong independent custodian where data may leave the premises. The on-premises constraint excludes it here. | | Removable disk cartridges | Rejected | Short shelf life and no write-once guarantee. | **What it buys** - No online credential reaches a vaulted cartridge - 30-year media life with a clear migration path - Cheapest cost per terabyte of any custodian **What it costs** - Restore from tape takes hours to days, and the stated RTO says so - Physical logistics and chain of custody - Media and drive generations must be managed **Choose differently when.** Where regulation allows data to leave the premises and there is no appetite for tape operations, a different organisation's object storage with Object Lock and its own credentials does the same job with a faster restore. **Why it holds up over time.** LTO has published a roadmap and kept read-back compatibility across generations since 2000. A copy that needs only a drive, Bareos and a key recorded in its envelope is about as long-lasting as digital storage currently gets. > **Lesson.** The second custodian's value is in how differently it fails, not in how fast it restores. ### Capture *How six kinds of engine are copied consistently, at the right version, without the platform reimplementing any of them.* #### ADR-10 · Engine-native tools behind one adapter contract and one envelope **Status:** Accepted · **Shown on views:** 07, 09, 12 *Do we build one backup mechanism for everything, or use each engine's own?* **Context.** The estate has six kinds of datastore. A universal mechanism (volume snapshots of everything) is uniform, but for a running database it produces crash-consistent copies whose restore is a recovery. A universal logical export is slow and loses point-in-time recovery. Every engine already has a mature, well-tested backup tool. **Decision.** Each engine class uses its own tool: pgBackRest, XtraBackup with binlog streaming, TiDB BR, ClickHouse BACKUP, Velero with Kopia for volumes and configuration exports, and the object mirror for buckets. Every adapter implements one contract (capture, write envelope, report size, duration, log position, checksum and consistency level) so that the catalogue, policy and verification layers see one kind of artefact. Tool-native expiry is disabled under capture identities. **How it works on-premise.** Adapters are thin Go wrappers packaged as sidecars or node agents that call the tool, stream the tool's output to custody, and write the envelope last. pgBackRest runs with repository encryption and automatic expiry off. Velero repository maintenance runs as a separate job under the disposal identity. The consistency level is taken from the adapter's declaration and the presence of freeze hooks, never inferred. | Option | Verdict | Reasoning | |---|---|---| | Native tools behind one contract | Chosen | Consistent, fast restores and point-in-time recovery from tools with years of production use, with differences contained in adapters. | | Storage snapshots for everything | Rejected | Crash-consistent copies of databases, and no PITR. | | Commercial multi-engine backup suite | Rejected | Conflicts with the open-source constraint, and its restore path is its own format. | | Logical exports for all databases | Right elsewhere | Good for small databases and version portability. Too slow for a 60-minute RTO on terabytes. | **What it buys** - Restore speed and PITR as good as each engine allows - Upstream fixes arrive without platform work - Differences stay inside adapters **What it costs** - Six tools to certify, upgrade and understand - Each tool's expiry and maintenance behaviour must be tamed under a no-delete identity - Operator skill across six tools **Choose differently when.** Consolidate on storage snapshots when every datastore runs on storage with application-consistent group snapshots and RPO is measured in hours rather than minutes. **Why it holds up over time.** Engine tools evolve with their engines, which is exactly the coupling needed. When an engine is retired or a better tool appears, only one adapter changes. Old artefacts keep the tool version in their envelope and remain restorable with that version. > **Lesson.** Standardise the contract, not the mechanism. #### ADR-11 · Point-in-time recovery is each engine's capability, orchestrated rather than reimplemented **Status:** Accepted · **Shown on views:** 09, 14 *Does the platform ship and replay its own change log, or rely on each engine's?* **Context.** One platform-owned log would give a uniform recovery promise, but it means capturing changes (logical decoding, binlog parsing, TiDB CDC) and writing replay for each engine, which amounts to reimplementing database recovery. Engine-native PITR is well tested but differs in granularity and limits between engines. **Decision.** Use WAL archiving with pgBackRest, binlog streaming with mysqlbinlog for MySQL, and BR log backup for TiDB. The platform standardises the recovery-point map (recovery_range with explicit gaps) and the promise per tier, and translates a requested timestamp into each engine's restore command. ClickHouse and volumes are declared as having no PITR. **How it works on-premise.** archive_timeout of 60 s with async push. A binlog streamer Deployment per MySQL primary uploading 1-minute chunks. A BR log task per TiDB cluster. Each adapter reports log ranges as it ships them. The catalogue marks a gap whenever a range does not join its predecessor, and the gap counts against RPO attainment. | Option | Verdict | Reasoning | |---|---|---| | Orchestrate native PITR with a uniform range map | Chosen | A uniform promise and report without owning replay correctness. | | Platform-owned CDC log and replay | Rejected | Reimplements recovery per engine and becomes the most dangerous code in the estate. | | No PITR; frequent snapshots only | Rejected | Cannot meet a 5-minute RPO or recover to just before a named event. | **What it buys** - Replay correctness is the engine's, tested by its community - Explicit gaps make RPO honest - Adding an engine means adding a range reporter **What it costs** - Granularity differs: to the second for PostgreSQL, to the event for MySQL, to the timestamp for TiDB - Replay speed is whatever the engine gives, which is why ADR-19 exists **Choose differently when.** Own the log only when many engines without native PITR need a minutes-scale RPO and restore into an environment the engines' tools cannot reach. **Why it holds up over time.** Engines keep improving their own recovery, and this design benefits from that automatically. The range map is a plain data structure that any future engine can report into. > **Lesson.** Coordinate what the experts already do well before deciding to do it yourself. #### ADR-12 · Refuse to capture with a tool and engine version pair that has not been certified by a restore **Status:** Accepted · **Shown on views:** 04, 20, 26 *How do we stop version drift from producing copies that write cleanly and do not restore?* **Context.** Version drift is the most common cause of a backup that succeeds and cannot be restored: XtraBackup behind a MySQL minor release, pgBackRest meeting a new WAL format, a BR version incompatible with its cluster. Nothing fails at capture time. It fails months later at restore. **Decision.** A nightly restore matrix certifies every engine version seen in the estate against every tool version in use at depth 3. Passing pairs are written to a signed file in Git. Every adapter checks its pair against the file before capturing and refuses if it is absent, which raises RPO-at-risk and pages the owner. Engine versions come from discovery, not declarations. **How it works on-premise.** GitLab CI triggers the matrix when Renovate proposes a tool upgrade or discovery reports a new engine version. The matrix runs in the enclave with synthetic datasets per engine and signs the result with cosign. Adapters verify the signature and read the pair list at start, and each capture reads the live engine version. | Option | Verdict | Reasoning | |---|---|---| | Signed certification gate, enforced at capture | Chosen | Drift becomes a same-day blocked capture instead of a failure found at restore. | | Pin versions by policy and trust teams | Rejected | The versions that break things are the unplanned ones. | | Rely on the daily rehearsal to catch drift | Rejected | Tier 2 and 3 rehearse too rarely, and every copy in between may be bad. | | Warn but capture anyway | Right elsewhere | Acceptable for Tier 3, where a blocked capture might cost more than a doubtful copy. Here it is not used even there, to keep one rule. | **What it buys** - No uncertified copy is written - Upgrades are proven by a restore before they matter - The certification list documents what works with what **What it costs** - An early engine upgrade blocks captures until certification, up to a day - Matrix runs use enclave capacity overnight - Synthetic datasets must cover each engine's features in use **Choose differently when.** Relax to warn-and-capture for a datastore whose owner accepts, in writing, that a doubtful copy is better than none during an upgrade window. **Why it holds up over time.** Engines and tools will keep releasing, faster rather than slower. A gate driven by what is actually running scales with that pace and needs no one to remember which versions are compatible. > **Lesson.** Turn a failure found late into a refusal made early. #### ADR-14 · Capture buckets from change notifications, reconciled by a weekly listing **Status:** Accepted · **Shown on views:** 08, 09 *How do we protect 1.3 PB of object storage without listing billions of objects every night?* **Context.** Tier 3 buckets change about 3% a day. A nightly listing to find changes costs time and gateway load in proportion to the total object count, not the change. Notifications are efficient but can be lost, and a missed notification is a silently unprotected object. **Decision.** Source buckets have versioning and persistent RGW bucket notifications to Kafka. The object mirror consumes them, copies each new version to custody with an envelope, and records the event. A weekly reconciliation lists each bucket and compares it with the catalogue, and any difference is copied and reported as a finding. **How it works on-premise.** Persistent notifications (queued by RGW until delivered) go to a three-broker Kafka topic per custody destination. The mirror is a Go consumer group that streams GET to PUT without staging and encrypts per object with the bucket's data key. Reconciliation shards listings by prefix and runs at low priority in the maintenance window. | Option | Verdict | Reasoning | |---|---|---| | Notifications with weekly reconciliation | Chosen | Work in proportion to change, plus a regular check that nothing was missed. | | Nightly rclone listing sync | Rejected | Listing cost grows with total objects and competes with Tier 1 for gateways. | | Storage-level replication to custody | Rejected | Replicates deletes and corruption, and joins the domains. | **What it buys** - Capture load follows change - Per-object envelopes allow per-object restore - Reconciliation limits the damage a lost notification can do to one week **What it costs** - Kafka becomes a dependency for Tier 3 RPO - Versioning on source buckets adds source storage - Many small artefacts in the catalogue **Choose differently when.** For buckets of large, rarely changing objects, a periodic listing is simpler and the notification pipeline is not worth running. **Why it holds up over time.** Change feeds plus periodic reconciliation is a general pattern for keeping two stores in step, used well beyond backup. Either part can change (a different broker, a different listing tool) without changing the other. > **Lesson.** Use an event stream for speed and a full comparison for truth, and run both. #### ADR-25 · Consistency groups by marker transactions, only for applications that cannot tolerate skew **Status:** Accepted · **Shown on views:** 09, 11 *How are several datastores restored to a mutually consistent point when their engines share no common quiesce?* **Context.** Two applications write to more than one datastore in one business operation (orders to PostgreSQL and TiDB, for example). Restoring each to the same wall-clock second does not guarantee consistency, because clocks differ and log positions do not line up. Coordinated freezes across engines would stall production. **Decision.** A consistency group writes a marker row with the same group id to each member datastore every 5 minutes through the application's own connection pool. Group restore recovers each member to the log position containing a chosen marker. Each group must declare whether operations in flight between markers are safe to replay, and supply a reconciliation check that the rehearsal runs. **How it works on-premise.** A small marker job per group, with credentials from OpenBao. Adapters index marker positions from WAL, binlog and TiDB log ranges into recovery_range. The group restore workflow restores members in parallel, each to the marker position, then runs the group's reconciliation assertion at depth 3. | Option | Verdict | Reasoning | |---|---|---| | Marker transactions with a tested reconciliation | Chosen | Consistency is approximate but aligned, and the remaining skew is checked, not assumed. | | Coordinated freeze across engines | Rejected | Stalls production writes and has no cross-engine primitive. | | Same-timestamp restore with no alignment | Rejected | Clock skew and commit ordering make it silently inconsistent. | | Groups for every multi-store service | Rejected | Most services tolerate skew. Groups only for the two that cannot. | **What it buys** - Group restore to an aligned point without stopping writes - Reconciliation is exercised by rehearsal - Cost limited to two applications **What it costs** - Marker rows in application schemas - Owners must write reconciliation checks - Five-minute alignment granularity **Choose differently when.** If an application moves to a single distributed database for both stores, the group becomes unnecessary and should be dissolved. **Why it holds up over time.** Markers written into each store's own log survive any change of backup tool, because they are data, not tool metadata. Multi-store consistency is an application property and stays with the application. > **Lesson.** When systems cannot agree on time, write the agreement into their data. ### Records *What the catalogue is, what it is not, and the evidence that must outlive the copies.* #### ADR-15 · Artefacts are authoritative and self-describing; the catalogue is a rebuildable index **Status:** Accepted · **Shown on views:** 10, 12, 26 *If the catalogue is lost, are the copies still usable?* **Context.** A catalogue that is the source of truth is fast and queryable, and it is a single point of failure for data it does not hold. At 30 million artefacts a scan is slow, but not prohibitively so. A rebuild that has never been run is exactly the kind of unproven claim this platform exists to eliminate. **Decision.** Every artefact has a signed envelope beside it in custody: engine and version, tool and version, consistency level, key id, chain parent, log range and checksum. The catalogue is derived. A full rebuild by scan runs every week into a shadow database and is diffed against the live catalogue. Restore can proceed from envelopes alone. **How it works on-premise.** Envelopes are JSON with a schema version, signed with the adapter's SPIFFE identity and written under the same Object Lock. Keys are prefixed by datastore, day and artefact ULID, so a scan can be split. The weekly job uses 32 listers and a projection builder writing to a standalone PostgreSQL instance, and a diff mismatch is a finding. | Option | Verdict | Reasoning | |---|---|---| | Self-describing artefacts with a weekly rebuild | Chosen | The index is an accelerator, and the rebuild is proven every week. | | Catalogue as the only source of truth, backed up | Rejected | The catalogue's backup becomes a special case with its own unproven restore. | | No catalogue; always scan | Rejected | Finding a recovery point would take hours, which breaks the RTO. | **What it buys** - Catalogue loss delays reports but does not prevent restores - Tools of the future can read today's copies - Weekly diff catches catalogue bugs **What it costs** - Envelope writing is part of every adapter - A few hours of custody listing load every week - Orphan artefacts without envelopes must be handled **Choose differently when.** If artefact counts reach billions, a full weekly scan stops being feasible. Move to incremental rebuild by day prefix and a monthly full scan. **Why it holds up over time.** A copy that carries its own description stays usable when the catalogue software, the schema and even the organisation that built them have all changed. The envelope schema is the one format here designed to last a decade, and it is versioned for that reason. > **Lesson.** Keep everything needed to restore next to the copy, and treat indexes as rebuildable. #### ADR-16 · The catalogue is an append-only event log in PostgreSQL with projections **Status:** Accepted · **Shown on views:** 07, 11 *How is catalogue state stored so that its history survives corrections?* **Context.** Artefact state changes: captured, copied, verified, held, expired. A mutable row loses what was claimed before a correction, which matters when auditing why a datastore was shown as PROVEN. The query load is modest: recovery-point lookups at p95 of 500 ms over 30 million artefacts. **Decision.** catalogue_event is the only table written to. Status, recovery ranges, custody copies and proofs are projections updated in the same transaction and can be dropped and replayed. PostgreSQL 17 with Patroni, a synchronous replica in DC-A and a standby cluster streaming to DC-B. **How it works on-premise.** Events are partitioned by month and never updated or deleted during retention. Projection tables are indexed on (datastore_id, from_ts) and (datastore_id, state). The catalogue itself is backed up with pgBackRest to custody, and is also independently rebuildable from envelopes (ADR-15). | Option | Verdict | Reasoning | |---|---|---| | PostgreSQL event table with projections | Chosen | Familiar, transactional and more than sufficient at this scale. | | Kafka as the event log with stream-processed views | Rejected | More moving parts for a workload with modest write rates. | | Mutable relational rows | Rejected | Loses the history of claims. | | ClickHouse | Right elsewhere | Good for analytics over proofs later. Wrong for transactional projections. | **What it buys** - Every past claim is reconstructable - Projections can be fixed by replay - Ordinary operational skills apply **What it costs** - Event growth must be partitioned and managed - Projection logic must be idempotent - Two recovery paths for the catalogue to keep healthy **Choose differently when.** Move to a distributed log only if many independent consumers need the event stream in real time, not just the catalogue. **Why it holds up over time.** An append-only log plus derived views has worked for decades, from accounting ledgers to database write-ahead logs. PostgreSQL has long-term community governance and an unusually conservative approach to compatibility. > **Lesson.** When history is evidence, record history and derive state from it. #### ADR-17 · Evidence is written before action, kept 7 years under a compliance lock, apart from the catalogue **Status:** Accepted · **Shown on views:** 10, 17, 25 *Where do audit events and drill records live, and what happens if they cannot be written?* **Context.** Every capture, restore, retention change, hold, disposal and access to a restored copy must be an attributed, immutable event written before the action completes. Drill records underpin every RTO claim and must outlive the artefacts they verified. If evidence shares the catalogue's database, it also shares its administrators and its failures. **Decision.** Destructive actions write an intent event first, and if that write fails the action does not start. Events are also sealed into hash-chained daily segments in an evidence bucket in custody B with compliance-mode Object Lock for 7 years. Verification and drill records follow the same path. **How it works on-premise.** The API writes intent to catalogue_event and to an outbox in the same transaction. A sealer service writes segments hourly and closes each day with a hash that includes the previous day's hash, and publishes the daily hash to the Git repository as an external anchor. Auditors verify the chain with a small open-source CLI. | Option | Verdict | Reasoning | |---|---|---| | Intent first, sealed segments under compliance lock | Chosen | No record, no action, and evidence nobody can change. | | Audit table in the catalogue only | Rejected | Catalogue administrators could alter it and its loss takes the evidence too. | | Log shipping to a SIEM only | Right elsewhere | Useful for detection, and events are also sent there. It is not the evidence store. | **What it buys** - Tamper-evident history with an external anchor - Evidence survives catalogue loss - Auditors verify independently **What it costs** - An evidence outage blocks destructive actions - A seven-year compliance lock cannot be undone if something is written by mistake - One more service **Choose differently when.** If a jurisdiction requires personal data in audit events to be erasable, store pseudonymous ids in sealed segments and keep the identity mapping in a separately erasable table. **Why it holds up over time.** Hash chains with an external anchor are a long-established method, and their verification needs only the data and a hash function. Seven-year evidence remains checkable long after this platform's software has been replaced. > **Lesson.** If an action matters enough to audit, it matters enough not to happen unaudited. ### Restore *How a copy becomes a running system inside its RTO, including the dangerous and the unknown cases.* #### ADR-18 · Side by side by default; in-place needs a second approver and a mandatory snapshot **Status:** Accepted · **Shown on views:** 05, 14, 17 *Who may restore, into what, and how is overwriting production made safe?* **Context.** Self-service restore brings RTO down and makes drills routine. Overwriting a production datastore with an older copy is the most destructive operation the platform can perform, and most recoveries do not need it: restoring side by side and copying back damaged rows keeps today's good writes. **Decision.** Owners may self-serve side-by-side, partial and isolated restores into targets whose classification is at least as strict as the source. In-place restore requires an approver from the owner group who is not the requester, using WebAuthn, an intent record, and a pre-restore snapshot of every target volume kept 7 days. A failed snapshot stops the workflow. **How it works on-premise.** OPA evaluates team ownership and target classification from NetBox and namespace labels. Keycloak provides WebAuthn step-up for approvals. The in-place workflow fences writers, takes Ceph RBD snapshots through the CSI snapshot API, restores, runs depth-2 checks and promotes. Reverting rolls back the snapshot. Bare-metal targets take a pgBackRest full copy instead. | Option | Verdict | Reasoning | |---|---|---| | Self-service side by side, gated in-place | Chosen | Fast for the common case, and deliberate and reversible for the dangerous one. | | Central team performs all restores by ticket | Rejected | Slow restores, and drills become rare. | | Full self-service including in-place | Rejected | One mistaken click overwrites today's data with last month's. | **What it buys** - Most restores need no second person - In-place restores are reversible for 7 days - Classification enforcement is automatic **What it costs** - Approver availability adds minutes to in-place restores - Snapshots need capacity on production storage - Owner metadata must be accurate for authorisation to work **Choose differently when.** Restrict self-service to a central team where target classification cannot be enforced automatically, such as unlabelled legacy environments. **Why it holds up over time.** Default to non-destructive and make the destructive path reversible: this holds however restores are implemented. The rules live in OPA policy, so they can change as the organisation changes without touching workflow code. > **Lesson.** Make the common recovery fast and the destructive one reversible. #### ADR-19 · RTO is a budget, and hourly block-incremental bases keep log replay inside it **Status:** Accepted · **Shown on views:** 05, 09, 14 *What actually determines whether a Tier 1 restore finishes in 60 minutes?* **Context.** Restore time is set by the slowest link. Transfer at 4 GB/s moves 4 TB in about 17 minutes, but PostgreSQL replays WAL on a single process. A day of WAL on a busy store can take over an hour to replay by itself. A base copy taken nightly makes the 60-minute RTO impossible for most of the day, whatever the storage throughput. **Decision.** The Tier 1 RTO is broken into a budget: 10 min to authorise and start, about 17 min to transfer, at most 10 min of replay, and 10 min of assertions and promotion. Hourly block-incremental bases bound replay distance. The 60-minute RTO applies to recovery points in the last 7 days, while points from day 8 to day 35 have a stated RTO of 3 hours. Every drill records all four parts separately. **How it works on-premise.** pgBackRest block incremental with bundling, run hourly and cheap because only changed blocks are copied. Weekly full, daily differential and hourly incremental, with chain depth up to 25 and at most 7 days from a full. Restores use process-max 32 across RGW gateways. Drill telemetry stores transfer, replay and assertion durations per run. | Option | Verdict | Reasoning | |---|---|---| | Budgeted RTO with hourly bases | Chosen | Makes the replay term, the usual limit, small and measured. | | Nightly base with WAL | Rejected | Replay alone can exceed the RTO late in the day. | | Warm standby restored continuously | Right elsewhere | The lowest RTO, at the cost of a second copy of every Tier 1 store. It is also close to being a replica, which cannot replace a backup. | **What it buys** - The RTO holds throughout the day, not only after the nightly copy - The four-part breakdown tells you what to fix when a drill is slow - Owners see the RTO step change at day 7 **What it costs** - Hourly capture load on Tier 1 sources - Longer backup chains - Two stated RTOs to explain **Choose differently when.** If replay speed improves enough (parallel recovery in future engine versions), hourly bases can become daily. The drill breakdown will show when that point is reached. **Why it holds up over time.** Whatever the technology, RTO is transfer plus replay plus checking. Measuring each part every day means the budget changes when the numbers change, instead of being inherited from a document. > **Lesson.** The part of recovery nobody benchmarks is usually the part that breaks the RTO. #### ADR-20 · Find the last clean recovery point by parallel nine-way search against an owner invariant **Status:** Accepted · **Shown on views:** 05, 16 *How do we recover to before a corruption whose start time nobody knows?* **Context.** Logical corruption is usually found weeks after it started. An SRE guessing recovery points one at a time spends a multi-hour restore per guess. The platform can restore any second, but only the owner can say what 'clean' means. **Decision.** Clean-point search takes an owner-supplied invariant and a window, restores 8 evenly spaced points in parallel in the enclave, evaluates the invariant, and narrows to the interval where it changes from clean to dirty. Seven rounds get from 35 days to one second. Every round is recorded as evidence. Tier 2 rehearsals pause during a search, and Tier 1 rehearsals continue. **How it works on-premise.** A Temporal workflow fans out 8 restores through the normal RestoreWorkflow, each to a point in the current interval, then runs the invariant through the assertion runner and chooses the next interval. Round duration is dominated by replay distance, which the hourly bases (ADR-19) keep short within 7 days. | Option | Verdict | Reasoning | |---|---|---| | Parallel k-ary search with owner invariant | Chosen | Turns days of guessing into hours of automated work with a record. | | Manual binary search by SRE | Rejected | Sequential, slow and unrecorded. | | Replay change log row by row looking for the bad write | Rejected | Needs application semantics the platform does not have. | **What it buys** - Recovery point found in about 3 hours inside the hourly window - Evidence of how the point was chosen - Reuses rehearsal capacity and code **What it costs** - Only as good as the invariant - Assumes corruption, once present, persists - Consumes 8 enclave slots for the duration **Choose differently when.** When corruption is intermittent or repaired and recurring, use the round results as input for a person rather than trusting the search's final answer. **Why it holds up over time.** Searching a monotonic predicate over time is a general algorithm. Faster restores make each round cheaper, and more slots make it wider. > **Lesson.** When you cannot know when something broke, make each check cheap enough to run many of them at once. #### ADR-21 · A manual restore path that needs only custody, the envelope, the key and the engine tool **Status:** Accepted · **Shown on views:** 08, 18, 26 *Can a restore be done when the platform itself is part of the outage?* **Context.** The day a restore is needed may be the day the control cluster, catalogue and console are down too, for example after a ransomware incident or a data-centre loss. A restore that needs the platform to be healthy has a hidden dependency on everything the platform depends on. **Decision.** A written runbook restores any engine class using only a break-glass custody read role, the envelope next to the artefact, OpenBao to unwrap the key, and the engine's tool at the version recorded in the envelope. It is rehearsed every quarter with the control cluster switched off, by someone who did not write it, and timed. **How it works on-premise.** Break-glass read credentials come from the sealed custody OpenBao after the usual two-officer unseal, from a read-only role kept separate from the bypass role. Tool container images at every certified version are mirrored to a registry at the vault site. A static binary of the bnr CLI lists envelopes by prefix and prints the exact restore command. | Option | Verdict | Reasoning | |---|---|---| | Documented, rehearsed, minimal-dependency runbook | Chosen | The restore does not depend on the platform being healthy. | | Highly available control plane as the answer | Rejected | Reduces the probability of the platform being down but not its correlation with the incident. | | Runbook written but not rehearsed | Rejected | An unrehearsed runbook is another unproven claim. | **What it buys** - Restore survives platform loss - The runbook's accuracy is proven quarterly - Engineers learn the underlying tools **What it costs** - Quarterly gameday time - The image mirror and CLI binary must be kept current - Break-glass read access is one more credential path to guard **Choose differently when.** Nothing flips this for Tier 1. For Tier 3, a manual path that is documented but only rehearsed yearly may be acceptable. **Why it holds up over time.** Because the dependencies are minimal and are the standard building blocks (an S3 API, a JSON file, a key manager, a tool), the runbook stays short when the platform gets complicated. It also becomes the migration path if the platform is ever replaced. > **Lesson.** Your last-resort restore must not depend on the system you are recovering from. ### Control and operations *What orchestrates the work, how protection is declared, and where everything runs.* #### ADR-22 · Temporal for durable workflows, with fail-static CronJobs for base copies **Status:** Accepted · **Shown on views:** 06, 07, 18 *What orchestrates captures, restores and rehearsals, and what happens when it is down?* **Context.** Restores and rehearsals are long, multi-step, wait on humans, retry throttled reads and must survive worker restarts. Cron can start a job but cannot resume a half-finished restore. Any orchestrator is also a dependency that can fail at the worst moment. **Decision.** Temporal orchestrates captures, restores, rehearsals and searches. Tier 1 log shipping runs outside it in the engines' own processes. Base copy schedules are also rendered as suspended Kubernetes CronJobs beside agents, and a watchdog unsuspends them when Temporal's heartbeat is missing for two intervals. Degradation order is reporting, console, Tier 3 capture, verification. Tier 1 capture and restore never degrade. **How it works on-premise.** Temporal server on the control cluster with PostgreSQL persistence (3 history, 2 frontend, 2 matching). Workers are Go. The policy compiler writes both Temporal Schedules and the CronJob manifests from the same compiled policy. The watchdog is a small DaemonSet on agent nodes. Envelopes written during fail-static mode are ingested when Temporal returns. | Option | Verdict | Reasoning | |---|---|---| | Temporal with fail-static fallback | Chosen | Durable, resumable workflows, without making capture depend on them. | | Argo Workflows | Right elsewhere | A good fit for batch pipelines. Weaker for long human-in-the-loop waits and signals. | | Kubernetes CronJobs only | Rejected | No resumable restores, approvals or in-flight ETA escalation. | | Custom state machine in the API | Rejected | Reimplements durable execution, badly. | **What it buys** - Restores survive worker crashes - Approvals are signals, not polling - Control-plane loss delays but does not lose Tier 1 **What it costs** - Temporal is a significant system to run - Two schedule representations to keep in step - Workflow code versioning discipline **Choose differently when.** For an estate of a dozen datastores with no human approvals, CronJobs and a simple restore script are proportionate. **Why it holds up over time.** Durable execution is a well-established category with several implementations. Workflows are ordinary Go code calling adapters, so a move to another engine rewrites orchestration, not adapters or custody. > **Lesson.** Use a workflow engine where it is needed, but do not put the most important path inside it. #### ADR-23 · Protection as policy on a declared tier, with no default and continuous discovery **Status:** Accepted · **Shown on views:** 03, 04, 21 *How does a new datastore get protected without someone remembering to set it up?* **Context.** Per-resource job configuration drifts and is forgotten for new resources. A default tier silently assigns an RPO nobody agreed. The biggest loss class is the datastore nobody enrolled, which a platform watching only its own jobs cannot see. **Decision.** Protection policy binds a resource class and a declared tier. Owners declare the tier by label in Git, the NetBox custom field or the namespace. A resource with no tier is refused where admission allows (Kyverno for PVCs and operator CRs) and reported everywhere else. Discovery runs every 15 minutes and raises an incident for anything unprotected for more than 72 hours. **How it works on-premise.** Policies are CRDs in Git, synced by Argo CD and compiled into schedules, retention and rehearsal slots. The coverage scanner reads NetBox, the Kubernetes API (PVCs, CloudNativePG and TiDB operator CRs), the RGW admin listing and OpenTofu state, and publishes unprotected age as a metric. Kyverno rejects a PVC without backup.tier in namespaces marked for production. | Option | Verdict | Reasoning | |---|---|---| | Tier-bound policy, no default, continuous discovery | Chosen | New resources inherit protection, and unknown ones become visible. | | Per-datastore jobs configured in a UI | Rejected | Drifts, and misses whatever nobody configured. | | Default everything to Tier 2 | Rejected | An RPO and a cost that nobody agreed to. | **What it buys** - Protection is reviewed like code - Unprotected resources have an age and an owner - New services get protection without asking **What it costs** - Admission refusals annoy teams at first - Inventory quality limits discovery - Resources outside NetBox and Kubernetes need other sources **Choose differently when.** Allow a temporary default with a short expiry during a large migration, when refusing every unlabelled resource would stop the migration. **Why it holds up over time.** Policy on classes rather than instances scales with the estate and survives inventory tools changing. Discovery sources are pluggable, and the rule that unprotected is a reported state, not an unknown one, stays the same. > **Lesson.** Make protection the default that must be declared, and make the absence of a declaration loud. #### ADR-26 · Two data centres and a vault site, a warm control plane, and a dedicated backup network **Status:** Accepted · **Shown on views:** 19, 26 *Where does everything run so that one site's loss leaves both copies and the ability to restore them?* **Context.** The platform must restore after losing a data centre, with its control plane possibly in that site. Backup and restore traffic runs at 3 to 4 GB/s and must not compete with production. The key service must keep quorum through a site loss. **Decision.** Custody A in DC-A, custody B and the enclave in DC-B, tape and one OpenBao voter at the vault site. The control plane is active in DC-A and warm in DC-B, with a Patroni standby catalogue and the Argo CD application defined but scaled to zero. Backup traffic runs on a dedicated 2 × 100 GbE network. A regional loss of both data centres is accepted as a tape-only recovery. **How it works on-premise.** OpenBao voters: 2 in DC-A, 2 in DC-B, 1 at the vault site. The Patroni standby cluster streams to DC-B. Starting the DC-B control plane is a runbook step (promote the standby, scale up) timed at gamedays. Capture and gateway hosts have dual 100 GbE ports on a separate spine from the service network. | Option | Verdict | Reasoning | |---|---|---| | Two data centres plus vault site, warm control | Chosen | Survives a site loss with both a copy and a way to restore it, at modest standing cost. | | Active-active control plane | Rejected | Adds Temporal multi-cluster complexity for a plane whose loss delays but does not lose data. | | Third data centre with a custody cluster | Deferred | Priced. Revisit if the regional risk is not accepted by the board. | | Backup traffic on the production network | Rejected | 3 GB/s of backup traffic becomes a production incident. | **What it buys** - Site loss survivable with measured RTO - Key service keeps quorum - No backup traffic on production links **What it costs** - Warm standby must be exercised to be real - Dedicated network hardware - Regional risk accepted, not removed **Choose differently when.** Add a third custody site when the regulatory set grows or the regional risk is no longer accepted. The copier pattern (ADR-08) extends to a third target without redesign. **Why it holds up over time.** Physical failure domains, independent networks and quorum across three sites are infrastructure basics that have not changed in decades. Each piece can be refreshed on its own hardware cycle without changing the topology. > **Lesson.** Place state so that each failure you accept still leaves a copy and a way to reach it. ### Identity and keys *Who may do what to which copy, and how cryptography enforces the separations that policy alone would not.* #### ADR-13 · Encrypt at the source with data keys wrapped by per-classification keys in OpenBao **Status:** Accepted · **Shown on views:** 08, 14, 18 *Who can read a copy, and how do we make a restore into the wrong environment fail?* **Context.** Custody administrators must never be able to read plaintext. Restores into a less protected environment must fail by construction. Lawful erasure must be possible across immutable copies. And a copy whose key is lost is not a backup. **Decision.** Adapters encrypt before bytes leave the source with a data key, per datastore per month for the regulatory set and per datastore per week otherwise. Data keys are wrapped by a key-encryption key per data classification in OpenBao Transit. Restore and verify identities may unwrap only for classifications their target is cleared for. Key versions are kept for longer than the longest retention and backed up offline under two-of-five escrow. **How it works on-premise.** OpenBao runs with 5 Raft voters across DC-A, DC-B and the vault site. Tools use their native encryption (pgBackRest repository cipher, xbstream encryption, BR crypter, ClickHouse encrypted disk, Kopia repository key), keyed by the unwrapped data key. The envelope records the key id, version and algorithm. Transit exports are wrapped to an offline key held in escrow, and restoring them is drilled every quarter. | Option | Verdict | Reasoning | |---|---|---| | Envelope encryption, per-classification wrapping keys | Chosen | Cryptographic separation of environments, cheap rotation, and erasure by destroying a key. | | Server-side encryption in custody | Rejected | Custody administrators could read plaintext, and a restore could go anywhere. | | One key per tool configured in files | Rejected | No classification separation, and rotation means re-encrypting data. | | HSM-backed key hierarchy | Right elsewhere | Adds tamper resistance where the regulator requires it, and can sit under OpenBao as a seal. | **What it buys** - A wrong-environment restore fails at unwrap - Rotation rewraps kilobytes, not petabytes - Erasure by destroying a key covers tape too **What it costs** - OpenBao is on the critical path for capture and restore - Tool-native encryption differs per engine and has to be certified - Offline key escrow has to be managed and drilled **Choose differently when.** If every engine and tool supported a common external key management protocol natively, move wrapping into that protocol and remove per-tool configuration. **Why it holds up over time.** Envelope encryption lets algorithms and key managers change without touching stored data: rewrap under a new key or a post-quantum wrapping algorithm, and the petabytes stay as they are. The envelope's key id and algorithm fields exist for that move. > **Lesson.** Put the separations you most need in key access, where a policy mistake cannot undo them. #### ADR-24 · Four disjoint custody roles assumed with SPIRE workload identity; no long-lived credentials **Status:** Accepted · **Shown on views:** 08, 24 *How are the four identities in the requirement made real, and what is stolen if a host is compromised?* **Context.** The requirement separates capture, custody administration, restore and verification with disjoint rights. Static S3 keys and database passwords on capture hosts are the credentials most likely to be stolen, and they last until someone notices. **Decision.** Custody has four RGW STS roles: capture (put and get, no delete, no retention change), custody admin (lock configuration, no object read), restore (get and extend retention) and verify (get only, from the enclave proxy). Workloads assume roles with SPIRE JWT-SVIDs. Source database credentials are issued per run by OpenBao's database engine. No long-lived credential exists in the platform. **How it works on-premise.** SPIRE server per data centre with the Kubernetes and node attestors. RGW is configured with SPIRE's OIDC discovery endpoint as a web-identity provider, and each role's trust policy matches a SPIFFE id path. STS sessions last 1 hour. OpenBao issues PostgreSQL and MySQL users with a 2-hour lease and a grant limited to backup functions. | Option | Verdict | Reasoning | |---|---|---| | STS roles via workload identity | Chosen | Nothing long-lived to steal, and rights per purpose. | | Static RGW keys in Kubernetes Secrets | Rejected | Long-lived and copyable, and often over-privileged. | | One service account for the platform | Rejected | A single compromise has every right, including the ones that must be separated. | **What it buys** - A stolen capture identity can write but never destroy - Credentials expire within hours - Rights are auditable per role **What it costs** - SPIRE becomes a trust root for custody roles - RGW STS web identity must be proven in the PoC - Database engines need backup-scoped roles designed per engine **Choose differently when.** Where SPIRE is not available, Kubernetes projected service-account tokens can be the web identity, at the cost of weaker node attestation. **Why it holds up over time.** Short-lived credentials from attested identity is where the whole industry is heading, and SPIFFE is a CNCF graduated standard. Roles defined by purpose remain valid whatever issues the tokens. > **Lesson.** Grant each job exactly the destruction it needs, which for capture is none. ## Every package used, in one table Every open-source package named on the views, what it is, and the job it does here. | Package | What it is | What it does here | Considered instead | |---|---|---|---| | Ceph (RGW) | Distributed storage system; RGW is its S3-compatible object gateway | Both custody clusters: Object Lock, versioning, STS roles, erasure-coded storage | MinIO, Garage, SeaweedFS | | OpenBao | Community fork of HashiCorp Vault under the Linux Foundation | Transit key wrapping, short-lived database credentials, and the sealed break-glass issuer | HashiCorp Vault, HSM | | Bareos | Network backup system descended from Bacula, with tape support | Writes Tier 1 and regulatory artefacts to LTO-9 WORM media at the vault site | Bacula, Amanda | | pgBackRest | PostgreSQL backup and restore tool | Full, differential and block-incremental base copies; asynchronous WAL archiving; parallel PITR restore | WAL-G, Barman | | Percona XtraBackup | Hot physical backup tool for MySQL | MySQL base copies, paired with mysqlbinlog streaming for PITR | MySQL Shell dump utilities | | TiDB BR | TiDB's Backup and Restore tool | Cluster-consistent snapshots and continuous log backup for point restore | Dumpling | | ClickHouse BACKUP | ClickHouse's native backup statement | Incremental table and partition backups to an encrypted S3 disk | clickhouse-backup | | Velero with Kopia | Kubernetes backup tool with a deduplicating, encrypted repository | Persistent volume snapshots moved to custody, and configuration exports | Stash, Kasten K10 (commercial) | | Apache Kafka | Distributed event log | Carries persistent RGW bucket notifications to the object mirror | Redpanda, NATS JetStream | | Temporal | Durable workflow execution engine | Orchestrates captures, restores, rehearsals, approvals and clean-point search | Argo Workflows, Cadence | | PostgreSQL with Patroni | Relational database with an HA manager | Catalogue event log and projections; Temporal persistence | CockroachDB, MySQL with Orchestrator | | SPIRE | Reference implementation of SPIFFE workload identity | Issues JWT-SVIDs that capture, restore and verify workloads exchange for RGW STS roles | Kubernetes projected service-account tokens | | Keycloak | Identity and access management server | Operator sign-in and WebAuthn step-up for approvals; a separate realm for custody | Authentik, Dex with an upstream IdP | | Open Policy Agent | General-purpose policy engine | Restore authorisation by ownership and target classification | Cedar, rules in code | | Kyverno | Kubernetes policy engine | Refuses PVCs and operator resources without a declared tier; guards enclave secrets | OPA Gatekeeper | | Cilium | eBPF-based Kubernetes networking and policy | Deny-all egress in the rehearsal enclave | Calico | | NetBox | Infrastructure source-of-truth application | Inventory and ownership of non-Kubernetes datastores for discovery | A CMDB | | Argo CD | GitOps continuous delivery for Kubernetes | Syncs policies, platform services and the warm DC-B control plane | Flux | | GitLab CE and Renovate | Source control with CI, and a dependency update bot | Runs the certification matrix when tools or engines change | Forgejo with Woodpecker | | cosign and Harbor | Artefact signing, and an OCI registry | Signed adapter images and signed certification lists; mirrored images at the vault site | Notation, Zot | | Healthchecks | Self-hostable dead-man's-switch monitoring service | Pages when rehearsal success pings stop arriving | Prometheus absent() rules | | Prometheus, Grafana, Loki, Alertmanager | Metrics, dashboards, logs and alert routing | RPO lag, proof age, coverage and showback, routed to on-call | VictoriaMetrics, OpenSearch |