Architecture One-Pager

Backup and Restore Service · Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises

A successful backup job proves that something was written. Only a restore proves that something can be recovered. This architecture is built around that difference.

Almost every serious data loss story has the same shape. There were several backup mechanisms, each reporting success, and none of them had ever been restored from until the day it was needed. The tool had drifted from the engine version, or the key had been rotated away, or the restore took nine hours against a one-hour promise, or the only copies sat in an account the attacker also controlled. The estate here is 120 datastores across six kinds of engine, about 1.4 PB, run in two of the organisation's own data centres. The job is not to take more copies. It is to be able to say, for each datastore, when a restore last actually worked and how long it took.

Each engine's own tool takes the copy (pgBackRest, XtraBackup, TiDB BR, ClickHouse BACKUP, Velero with Kopia) behind one adapter contract. A version gate refuses any tool and engine pair that has not been certified by a real restore. Copies are encrypted at the source and written to Ceph object storage with Object Lock, in a custody domain the backup platform cannot delete from. A verifying copier fills a second, independently administered cluster in the other data centre, and Tier 1 and the regulatory set also go to WORM tape at a third site. A PostgreSQL catalogue indexes it all, but every copy describes itself, so the catalogue can be rebuilt by scanning. Every day, an isolated cluster with no route out restores each Tier 1 datastore through the same Temporal workflow an emergency would use. It checks structure, content and whether the application boots, and records a proof with a measured duration. Protection status is computed from those proofs and nothing else, and a monitor run by another team pages when the proofs stop arriving.

What it is, and what it is not

• A platform that proves every datastore can be restored, with a date — not a backup scheduler with a green dashboard of job results.
• Custody the backup platform can write to but cannot delete from — not immutability implemented in the platform's own code, which a stolen credential would bypass.
• Each engine's own backup and point-in-time tooling, orchestrated — not a home-grown universal backup format that reimplements database recovery.
• A catalogue that makes restores fast — not a single point of failure. The copies describe themselves and the catalogue is rebuilt from them every week.
• An RTO measured in drills and published next to the target — not an RTO calculated from a storage vendor's throughput sheet.
• Protection against the data being corrupted, deleted or encrypted — not high availability. Replicas and failover belong to the protected systems, and a replica is never counted as a backup.

The decisions that are the architecture

1. Status is a verification outcome (ADR-01) — Only a successful rehearsal writes PROVEN, and proof expires at twice the verification interval. The field that is reported, paged on and audited is last_proven_restore.
2. Rehearse through the real path (ADR-03) — Rehearsals use the same workflow, adapters, keys and custody reads as an emergency, to a random second, alternating custody copies. A special verification path would only prove itself.
3. Watch for silence from outside (ADR-04) — Healthchecks, run by the observability team, expects a success ping per tier and per Tier 1 datastore, and pages at twice the interval. A pipeline that stops looks exactly like a healthy one to a monitor that only watches for failures.
4. A custody domain the platform cannot delete from (ADR-06) — Ceph RGW with Object Lock set at write time, on separate racks with separate administrators and credentials. The capture role can put and get, and nothing else.
5. Two people, enforced by cryptography (ADR-07) — The permission to shorten a lock lives only in a sealed OpenBao instance that needs 2 of 5 officer key shares to open. It issues one scoped 15-minute credential per unseal, and no standing identity holds that permission.
6. A second custodian that fails differently (ADR-09) — WORM tape written by Bareos and vaulted at a third site, for Tier 1 and the regulatory set, with its own quarterly restore rehearsal.
7. Engine-native tools behind one contract (ADR-10) — Six mature engine tools do the byte-level work and each engine's own point-in-time recovery. The platform standardises the artefact envelope, the recovery-point map and the promise.
8. Uncertified version pairs cannot capture (ADR-12) — A nightly restore matrix certifies each engine version against each tool version at depth 3. Adapters refuse any pair not in the signed list, so drift shows up as a blocked capture that day, not a failed restore months later.
9. Artefacts are the truth; the catalogue is an index (ADR-15) — A signed envelope beside every artefact carries everything a restore needs. The catalogue is rebuilt from a scan every week and diffed against the live one.
10. RTO is a budget, and replay is the line item people forget (ADR-19) — Hourly block-incremental bases keep WAL replay under an hour for Tier 1. The 60-minute RTO is allocated to authorisation, transfer, replay and assertions, and each part is measured every day.

Why this holds up over time

Backup platforms usually age badly in one of three ways. The tooling is replaced and nobody can read the old copies. A vendor or licence changes and the custody layer has to be migrated under pressure. Or the drills quietly stop, and the platform is back to reporting job success. The design defends against each of these on purpose.

• The central property belongs to no technology. A copy counts once a restore from it has worked. That was true of tape in 1990 and will be true of whatever replaces object storage. The verifier, the proof record and the silence alarm are the lasting parts, and none of them assumes a particular engine, tool or storage product.
• Every tool sits behind a contract that outlives it. Adapters are replaceable one engine at a time. The envelope is a versioned, documented JSON schema stored next to tool-native data, so a copy written by pgBackRest in 2026 can still be found, described and restored with the pgBackRest version recorded in it, even after the platform has moved to another tool.
• Custody speaks the most widely implemented storage API. The platform uses nothing from Ceph except the S3 API, Object Lock and STS. Any S3-compatible store with Object Lock can replace it, and moving is a copy under a new lock, not a redesign. Nothing depends on the Ceph admin API, deliberately.
• Open source chosen for its governance, not just its licence. Recent relicensing forced forks and migrations elsewhere in the industry: HashiCorp Vault moved to the Business Source License in 2023, and MinIO's community edition narrowed sharply in 2025. Custody and keys therefore sit on foundation-governed or long-stable projects: Ceph, PostgreSQL, and OpenBao under the Linux Foundation. Every choice has a named alternative in the decision record.
• Keys can change without rewriting a petabyte. Artefacts are encrypted with data keys wrapped by per-classification key-encryption keys in OpenBao, and the envelope records the key id and algorithm. Rotating a key, or moving to a new algorithm, rewraps small data keys and leaves the stored bytes alone. Old key versions are kept for longer than the longest retention.
• Numbers are re-measured, not inherited. RTO, restore throughput and replay speed are measured again every day, so when hardware, data volume or engine versions change, the published numbers change too. A design sized once, on paper, goes out of date without anyone noticing. This one shows its drift in the daily report.
• Controls that survive people leaving. Two-person rules are enforced with key shares that are re-issued whenever a holder leaves, not by remembering who is trusted. Runbooks are proven by gamedays run by people who did not write them, so knowledge does not leave with the original team.

Non-functional targets

Targets from the requirement, and the mechanism that meets each one. Every duration and throughput figure is replaced by a measured value at the first drill and republished monthly.

Scope

In scope

• Capture, custody, catalogue, restore and verification for about 120 datastores in two data centres
• Point-in-time recovery for Tier 1 PostgreSQL, MySQL and TiDB
• Isolated rehearsal at a declared depth per tier, with measured RTO
• Two-person control of retention reduction, legal hold and disposal
• WORM tape as the second custodian for Tier 1 and the regulatory set
• Coverage discovery and the unprotected-resource report
• Cost showback per team, with verification on its own line

Explicitly out of scope

• High availability, replication and site failover of the protected systems
• Archival of data for analytical reuse, and data export for portability
• Endpoint, laptop and SaaS application backup
• Business reconciliation of rows written after a corruption, which belongs to the owning team
• Protection against a regional disaster taking both data centres beyond tape, which is accepted and registered as a risk

The six-week proof

Not a slice of every feature. The proof shows the properties everything else depends on: a copy the platform cannot delete, a restore that works with the platform switched off, and a proof record that shows the silence alarm working.

1. Two small Ceph clusters with Object Lock and STS; capture role denied delete and bypass, confirmed by attempting both
2. pgBackRest against one PostgreSQL cluster with hourly block-incremental bases and WAL archive-async to custody A
3. Temporal restore workflow into an isolated namespace to a random second, with a depth-3 invariant and a measured duration
4. Healthchecks dead-man check with the rehearsal pipeline deliberately stopped, and time-to-page recorded
5. Sealed OpenBao with 2-of-5 shares issuing one scoped bypass credential, then resealed

• Delete the catalogue database, rebuild it by scanning envelopes, and restore from the rebuilt copy
• Run the manual restore runbook with the control cluster switched off, performed by someone who did not write it
• Upgrade PostgreSQL by one minor version without certification and confirm the capture is refused and paged
• Steal the capture credential in a test and confirm every delete, overwrite of a locked version and retention change is refused

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 26 records across 7 areas, each with the alternatives that lost and what the choice costs.