Architecture Views

What event makes the platform say a datastore is protected?

Context

Capture jobs report success when bytes are written. A copy can be written cleanly and still be unrestorable: the wrong engine version, a truncated chain, a lost key, or a restore far slower than the RTO. Most of the loss this platform exists to prevent looked fully protected on a job-status dashboard.

Decision

A datastore is PROVEN only when a rehearsal restored it at its declared depth inside the verification interval. Proof expires at twice that interval. Capture success moves a datastore to Captured, which is displayed as unproven. last_backup_succeeded does not appear in the status projection at all.

How it works on-premise

The verifier appends a verified event carrying depth, assertion counts, measured duration and custody location to catalogue_event in PostgreSQL. protection_status is a projection maintained in the same transaction. A Prometheus SQL exporter publishes the age of last_proven_restore, and Alertmanager pages the owner when it passes twice the interval.

Options weighed

• ChosenVerifier-written status with expiring proof: The only option where the thing reported is the thing that matters on the day of need.
• RejectedJob success as status, rehearsal as a separate report: Two reports, and in practice everyone reads the green one.
• RejectedJob success plus periodic checksum verification: Proves bytes are intact, not that a system comes back. It misses version drift, key loss and slow restores.
• Right elsewhereStatus from job success for low tiers only: Reasonable for scratch data with no RTO. Here, Tier 3 still gets sampled proof, because it is 1.3 PB of someone's data.

Consequences

What it buys

• The dashboard and the day of need agree
• Rehearsal failures surface as normal, owned work
• Auditors get a date and a measured duration

What it costs

• A rehearsal pipeline must exist before anything is called protected
• Status looks worse than job-success dashboards at launch, which needs managing
• Verification becomes the largest recurring cost

Choose differently when

Choose job-based status only for data whose loss the business has explicitly accepted, such as caches and reproducible build outputs, and label that status 'captured' rather than 'protected'.

Why it holds up over time

It does not matter which engine, tool or storage medium is in use: a copy is only as good as the last restore from it. Tools will change several times over the life of this platform, and this rule will not need to.

What does 'verified' actually mean, and should it mean the same thing for every datastore?

Context

Each extra depth of assertion catches a different failure and costs roughly an order of magnitude more. A restore that completes catches unreadable copies. Structure checks catch partial restores. Content checks catch silent loss. Booting the application catches copies that are intact but unusable. Running depth 4 for 120 datastores daily would need a second production estate.

Decision

Tier 1 is rehearsed daily at depth 4, Tier 2 on a rotation at depth 3 with every datastore proven within 30 days, and Tier 3 by 0.1% daily sampled read-back at depth 2 plus one full bucket restore per quarter at depth 3. The depth actually reached is recorded on every proof. Lowering depth or frequency requires the tier's RPO and RTO owner to accept it in writing.

How it works on-premise

The assertion runner is a container that runs engine-specific checks. Depth 2 checks the schema migration version, object counts and partition coverage. Depth 3 runs owner-supplied invariant SQL, counts within tolerance, and a checksum over sampled cold key ranges compared to a probe taken on the source at a recorded log position. Depth 4 deploys the owner's Helm chart into the enclave against the restored copy and runs their smoke-suite container.

Options weighed

• ChosenDepth per tier, depth recorded per proof: Spends verification money where the RTO is short and the data is critical.
• RejectedDepth 1 everywhere, daily: Cheap, and blind to most of the failures that matter.
• RejectedDepth 4 everywhere, weekly: Unaffordable at this estate size, and the gap is too long for Tier 1.
• Right elsewhereStatistical sampling across all tiers: Right for very large fleets of identical datastores. This estate has few, dissimilar ones.

Consequences

What it buys

• Cost is proportional to criticality and argued in one place
• Each depth maps to named failures, so a gap is visible
• Owners can see what a cheaper tier gives up

What it costs

• Depth 3 and 4 need owners to write invariants and smoke suites
• Tier 3 proof is probabilistic and has to be explained with its confidence figure
• Depth 4 needs a deployable application in an isolated network

Choose differently when

Raise Tier 2 to depth 4 if a real incident shows an intact-but-unusable copy there. Drop Tier 1 to depth 3 for a datastore whose application genuinely cannot boot in isolation, with the gap written against the tier.

Why it holds up over time

The failure classes (unreadable, incomplete, silently wrong, unusable) are properties of data and applications, not of any tool. As hardware gets cheaper, the per-tier depths can be raised without changing the structure.

Should verification run through its own optimised path, or through exactly the path a real restore would use?

Context

It is tempting to verify by reading a snapshot directly or mounting a volume, which is fast and cheap. But a verification path that differs from the emergency path proves only the verification path. Restores to 'now' never exercise long replay, and restores from the nearest custody cluster never measure the cross-site one.

Decision

Rehearsals start the same Temporal RestoreWorkflow as an emergency, with the same adapters, identities, key unwrap and custody reads. Only the target (the enclave) and a purpose field differ. Tier 1 targets a random second in the last 24 hours. Rehearsals alternate between custody A and custody B. Durations count toward published RTO attainment.

How it works on-premise

The verifier schedules Temporal workflows with purpose=drill and target=enclave. The restore activities are the same code as for emergencies, and a CI rule fails any change that adds a purpose branch outside target selection. Custody location is recorded on the proof, and the enclave in DC-B reads A over the inter-site link.

Options weighed

• ChosenSame path, random point, alternating copy: Every drill is a real restore, so its duration is a real RTO sample.
• RejectedDedicated fast verification path: Proves the fast path works, which nobody will use in an incident.
• RejectedRestore to latest point only: Never exercises the replay that old recovery points need.
• RejectedVerify in production against a live replica: Puts a copy of production and restore traffic inside the production network, and proves nothing about custody.

Consequences

What it buys

• Measured RTO is a byproduct of verification
• Regressions in the restore path show up the next morning
• Cross-site restore throughput is known, not assumed

What it costs

• Rehearsals are as slow as real restores
• Custody reads across sites cost link capacity every other day
• Restore path changes need drill capacity to test

Choose differently when

A separate path is reasonable where the restore procedure is purely manual and cannot be automated, but then the drill is a timed human exercise, and it should be run more often.

Why it holds up over time

However the restore path changes, the drill follows it automatically because it is the same code. Nothing needs to be kept in step by hand, and code that has to be kept in step by hand eventually drifts.

How do we find out that verification has quietly stopped?

Context

The most dangerous failure of a verification pipeline is not a failed rehearsal, which pages. It is the pipeline not running: a suspended schedule, an expired credential, or a notification route dropped upstream. To a monitor that watches only for failures, all of these look like health. A monitor inside the platform fails along with it.

Decision

Every successful rehearsal pings a Healthchecks check, one per tier and one per Tier 1 datastore, with a grace period of twice the interval. Healthchecks is operated by the observability team on the shared estate and pages through their routes. Silence is a page, not a ticket.

How it works on-premise

A self-hosted Healthchecks instance (BSD licence) runs on the observability team's cluster in DC-B, backed by their PostgreSQL. The verifier calls it with a plain HTTPS GET to the check's URL after writing the proof, with no platform identity involved. A Prometheus blackbox probe from the same team monitors Healthchecks itself.

Options weighed

• ChosenPush-based dead-man check operated by another team: Independent failure domain, and alerts on absence by design.
• RejectedPrometheus absent() or timestamp-age rules in our own stack: Fails with the platform's own scrape path and is easy to silence with the platform's own access.
• RejectedDaily email report of rehearsals run: Nobody notices the email that did not arrive.
• Right elsewhereA commercial SaaS dead-man service: A fine choice where outbound internet from the monitoring estate is acceptable. It is excluded by the on-premises constraint here.

Consequences

What it buys

• A stopped pipeline pages within two intervals
• The alarm does not share credentials or infrastructure with the thing it watches
• One more organisation has visibility of recoverability

What it costs

• A cross-team dependency with its own change process
• Check lists must be kept in sync as datastores are enrolled
• A monitor to be monitored

Choose differently when

If there is no independent observability team, host the check at the vault site on separate hardware, run by custody officers. What matters is that it is independent, not which team owns it.

Why it holds up over time

Alerting on absence is a pattern that does not depend on any tool. Healthchecks can be replaced by any push-based check without the verifier changing more than a URL.

Where do daily restores of full production data go?

Context

Rehearsals restore full copies of production every day into the environment that is created most often and watched least. A namespace in a shared cluster is one bad network policy away from connecting a restored payments database to something that can reach it, or reach out from it.

Decision

A dedicated Kubernetes cluster in its own VRF in DC-B, with deny-all egress at both the network and the Cilium layer, no production credentials, and a single read-only proxy to custody. Restored data is destroyed at the end of each rehearsal, and access to a restored copy is an audited event. Depth-4 applications run with enclave-only configuration.

How it works on-premise

16 NVMe nodes. The data-centre firewall permits the VRF to reach only the custody read proxy and the internal registry mirror. CiliumNetworkPolicy defaults to deny with a cluster-wide rule. Kyverno rejects Secrets that reference production credential paths. Volumes are provisioned from local NVMe and wiped with blkdiscard when released.

Options weighed

• ChosenDedicated cluster in its own VRF: A network boundary enforced below Kubernetes, so a misconfiguration inside the cluster cannot open a route out.
• RejectedNamespace in a shared non-production cluster: A label is not a boundary for data this sensitive.
• RejectedRestore into the production cluster: Puts restored data one DNS name away from real clients.
• Right elsewhereEphemeral virtual machines per rehearsal: Good isolation, but slower to start for depth-4 application boots at this cadence.

Consequences

What it buys

• Rehearsal cannot leak data or send to real downstream systems
• Security review happens once, for a fixed environment
• Wiping is automatic

What it costs

• A second cluster to run and patch
• Depth-4 smoke suites cannot call real dependencies and need stubs
• An image registry mirror inside the enclave

Choose differently when

If masking obligations require it, run content assertions on masked copies. In that case the enclave's isolation matters less than the masking pipeline, and that pipeline must be rehearsed too.

Why it holds up over time

The risk (full copies of production in an environment used every day) grows with every new datastore, not with any particular technology. A network boundary below the orchestrator remains valid whether rehearsal later moves to virtual machines or to something else.

Where do copies live so that nothing able to destroy the original can reach them?

Context

The copies must survive a compromised backup platform, a stolen credential and a well-meaning operator having a bad day. Immutability implemented in application code protects only against callers who use that code. The requirement puts the failure domain at the account boundary, which on-premises means a separate administrative domain: hardware, credentials and staff.

Decision

Artefacts are written to Ceph RGW buckets with versioning and Object Lock default retention applied at write time, on racks, switches, bastions and a Keycloak realm that the backup platform and production administrators cannot access. The platform reaches custody only through the S3 API with STS roles. Custody root access is two-person.

How it works on-premise

Custody A has 18 hosts of 24 × 22 TB HDD with erasure coding 8+3 and 8 RGW gateways behind HAProxy on the backup network. Bucket defaults set the retention mode and period per tier. Versioning means a tool rewriting a manifest creates a new version and the locked version remains. Host root access requires SSH certificates issued only against two approvals, and RADOS-level object operations outside RGW raise an alert.

Options weighed

• ChosenCeph RGW with Object Lock in a separate domain: Storage-enforced immutability through the most common storage API, on a project with stable governance.
• RejectedMinIO with Object Lock: Technically capable, but the community edition's scope and distribution changed during 2025, which is the wrong profile for a store meant to last a decade.
• RejectedBackup appliance with a proprietary immutable store: Conflicts with the open-source constraint, and its restore path depends on one vendor's software.
• Right elsewhereDeduplicating backup server on a filesystem (Bareos disk, restic REST): Simpler for small estates. The immutability is weaker and it does not scale to 5 PB.

Consequences

What it buys

• A stolen capture credential cannot delete or overwrite a locked version
• Custody can be replaced by any S3 Object Lock implementation
• Erasure coding keeps the raw-to-usable ratio at 1.375

What it costs

• Ceph needs skilled operators, and the custody team must include them
• Object Lock protects against S3 callers, not against storage root
• Versioned buckets grow with every manifest rewrite until expiry

Choose differently when

For estates under about 100 TB, a pair of well-separated backup servers with append-only repositories and tape may be simpler and adequate. Above a few petabytes with a strong compliance mandate, adding a third-site cluster is the next step.

Why it holds up over time

The S3 API with Object Lock is the most widely implemented form of object storage, available from several independent open-source and commercial implementations. Because the platform uses only that API, custody can move to another implementation without redesign. Separating administrative domains is an organisational control and does not depend on the storage technology.

Who can shorten a retention lock, release a legal hold, or delete before expiry, and how do we make sure that is never one person?

Context

Compliance-mode locks cannot be shortened by anyone, which makes lawful erasure and correcting a wrong tier impossible for up to seven years. Governance mode can be bypassed by any caller with the bypass permission, so whoever holds that permission is a single point of destruction. A workflow that asks for two approvals but then uses one service credential only moves that single point somewhere else.

Decision

Tier retention uses governance mode. The bypass permission is held by no standing identity. It is issued by a dedicated OpenBao instance in the custody domain that stays sealed. Unsealing needs 2 of 5 Shamir shares held by custody officers, neither of whom may be the requester. Once unsealed, it issues one 15-minute credential scoped to the listed object versions, and it is sealed again straight away. The evidence store uses compliance mode.

How it works on-premise

The break-glass OpenBao runs on two small hosts in the custody racks with integrated storage and auto-unseal disabled. The retention broker (Go) records intent with the object-list hash, collects approvals through Keycloak WebAuthn, waits for the unseal, requests the credential from a role whose policy is templated to the object list, performs PutObjectRetention with the bypass header, then calls seal. Placing a hold or extending retention needs none of this.

Options weighed

• ChosenGovernance mode plus threshold-sealed issuer: Two people are needed by construction, and disposal is still possible when the law requires it.
• RejectedCompliance mode for everything: A mistaken seven-year lock cannot be released, and erasure obligations cannot be met.
• RejectedApproval workflow holding a standing bypass key: Whoever compromises the workflow's host gets the bypass, with or without approvals.
• Right elsewhereHSM with an m-of-n card quorum: Equivalent strength, better physical tamper resistance, and a higher cost and procurement burden.

Consequences

What it buys

• No person or component alone can reduce retention
• The mechanism is an audited open-source primitive, not new cryptography
• Legal disposal remains possible

What it costs

• Every reduction takes about an hour of two officers' time
• Share custody must be kept up through staff turnover
• If three of five holders are unavailable, reductions stop until shares are re-issued

Choose differently when

If the regulator requires records that cannot be altered at all and there is no erasure obligation, use compliance mode for that set and accept that a mistake lasts until expiry.

Why it holds up over time

Threshold secret sharing is decades old and does not depend on a product. The design depends only on 'sealed until k of n agree', which any successor key manager or HSM provides. Share re-issue on role change keeps the control valid as people come and go.

How does a Tier 1 copy reach the second data centre?

Context

The requirement asks for Tier 1 in two locations. Ceph's multisite replication does this natively, but it joins both clusters into one realm with shared metadata and an administrator who can act on both. It also copies whatever arrives, including a corrupted object, without checking it.

Decision

Custody A and custody B are separate Ceph clusters with separate realms, keys and administrators. A custody copier with a read role on A and a write-only role on B copies each artefact, recomputes SHA-256 against the envelope, and only then records a copied event. Tier 3 goes to the custody cluster in the data centre opposite its source.

How it works on-premise

The copier is a Go service driven by captured events from the catalogue, with a lag target under 15 minutes for Tier 1. It holds two STS sessions, one per cluster, from different trust configurations. A checksum mismatch is a durable finding and the copy is retried from A. B's bucket defaults set their own Object Lock retention independently of A.

Options weighed

• ChosenIndependent clusters with a verifying copier: Two administrative domains, and the second copy is checked before it counts.
• RejectedRGW multisite zone replication: One realm and one blast radius, and nothing is verified end to end.
• RejectedCapture agents write to both clusters: Doubles source-side load and puts write access to both custodians on every source host.
• Right elsewhererclone sync on a schedule: Fine for small estates, but it lists everything and has no event record per artefact.

Consequences

What it buys

• Compromising custody A's administration does not reach B
• Every second copy has been checked against its checksum
• Copy lag is a monitored number per tier

What it costs

• Own code to write and run
• The copier's two roles need careful review
• Twice the Ceph operational surface

Choose differently when

Use native multisite where both sites really are administered by one team under one risk model and speed of replication matters more than independence.

Why it holds up over time

Copy, verify, then count is independent of storage technology. If either cluster is replaced, the copier changes one endpoint.

What takes the role of the off-provider copy for an estate that is already on-premises?

Context

In the cloud version of this requirement, the second custodian protects against losing the provider or the account. On-premises, the equivalent common failure is the storage software and its administrators: a Ceph bug, a compromised Ceph administrator, or one team's mistake across both clusters. The second custodian must not share any of those.

Decision

Tier 1 weekly fulls and the regulatory set's monthly copies are written by Bareos to LTO-9 WORM cartridges, vaulted at a third site under custody officers' control. Once tape is live, one Tier 1 datastore is restored from tape every quarter in the enclave and timed. Writing to tape starts in Phase 3, as the requirement stages it, but the library is ordered during the MVP because tape hardware takes the longest to procure.

How it works on-premise

A tape library with 4 LTO-9 drives at the vault site, a Bareos director and storage daemon on separate hosts, and a staging pool that reads already-encrypted artefacts from custody B with a read role. Cartridges are write-once media and leave the library on a weekly rotation. Media is migrated every two LTO generations, and a working drive of the old generation is kept until migration is proven by a restore.

Options weighed

• ChosenWORM tape via Bareos, vaulted: Different media, software and administrators, and offline once vaulted.
• RejectedThird Ceph cluster at the vault site: Shares the software, bugs and skills of the first two.
• Right elsewhereCommercial cloud object storage with Object Lock: A strong independent custodian where data may leave the premises. The on-premises constraint excludes it here.
• RejectedRemovable disk cartridges: Short shelf life and no write-once guarantee.

Consequences

What it buys

• No online credential reaches a vaulted cartridge
• 30-year media life with a clear migration path
• Cheapest cost per terabyte of any custodian

What it costs

• Restore from tape takes hours to days, and the stated RTO says so
• Physical logistics and chain of custody
• Media and drive generations must be managed

Choose differently when

Where regulation allows data to leave the premises and there is no appetite for tape operations, a different organisation's object storage with Object Lock and its own credentials does the same job with a faster restore.

Why it holds up over time

LTO has published a roadmap and kept read-back compatibility across generations since 2000. A copy that needs only a drive, Bareos and a key recorded in its envelope is about as long-lasting as digital storage currently gets.

Do we build one backup mechanism for everything, or use each engine's own?

Context

The estate has six kinds of datastore. A universal mechanism (volume snapshots of everything) is uniform, but for a running database it produces crash-consistent copies whose restore is a recovery. A universal logical export is slow and loses point-in-time recovery. Every engine already has a mature, well-tested backup tool.

Decision

Each engine class uses its own tool: pgBackRest, XtraBackup with binlog streaming, TiDB BR, ClickHouse BACKUP, Velero with Kopia for volumes and configuration exports, and the object mirror for buckets. Every adapter implements one contract (capture, write envelope, report size, duration, log position, checksum and consistency level) so that the catalogue, policy and verification layers see one kind of artefact. Tool-native expiry is disabled under capture identities.

How it works on-premise

Adapters are thin Go wrappers packaged as sidecars or node agents that call the tool, stream the tool's output to custody, and write the envelope last. pgBackRest runs with repository encryption and automatic expiry off. Velero repository maintenance runs as a separate job under the disposal identity. The consistency level is taken from the adapter's declaration and the presence of freeze hooks, never inferred.

Options weighed

• ChosenNative tools behind one contract: Consistent, fast restores and point-in-time recovery from tools with years of production use, with differences contained in adapters.
• RejectedStorage snapshots for everything: Crash-consistent copies of databases, and no PITR.
• RejectedCommercial multi-engine backup suite: Conflicts with the open-source constraint, and its restore path is its own format.
• Right elsewhereLogical exports for all databases: Good for small databases and version portability. Too slow for a 60-minute RTO on terabytes.

Consequences

What it buys

• Restore speed and PITR as good as each engine allows
• Upstream fixes arrive without platform work
• Differences stay inside adapters

What it costs

• Six tools to certify, upgrade and understand
• Each tool's expiry and maintenance behaviour must be tamed under a no-delete identity
• Operator skill across six tools

Choose differently when

Consolidate on storage snapshots when every datastore runs on storage with application-consistent group snapshots and RPO is measured in hours rather than minutes.

Why it holds up over time

Engine tools evolve with their engines, which is exactly the coupling needed. When an engine is retired or a better tool appears, only one adapter changes. Old artefacts keep the tool version in their envelope and remain restorable with that version.

Does the platform ship and replay its own change log, or rely on each engine's?

Context

One platform-owned log would give a uniform recovery promise, but it means capturing changes (logical decoding, binlog parsing, TiDB CDC) and writing replay for each engine, which amounts to reimplementing database recovery. Engine-native PITR is well tested but differs in granularity and limits between engines.

Decision

Use WAL archiving with pgBackRest, binlog streaming with mysqlbinlog for MySQL, and BR log backup for TiDB. The platform standardises the recovery-point map (recovery_range with explicit gaps) and the promise per tier, and translates a requested timestamp into each engine's restore command. ClickHouse and volumes are declared as having no PITR.

How it works on-premise

archive_timeout of 60 s with async push. A binlog streamer Deployment per MySQL primary uploading 1-minute chunks. A BR log task per TiDB cluster. Each adapter reports log ranges as it ships them. The catalogue marks a gap whenever a range does not join its predecessor, and the gap counts against RPO attainment.

Options weighed

• ChosenOrchestrate native PITR with a uniform range map: A uniform promise and report without owning replay correctness.
• RejectedPlatform-owned CDC log and replay: Reimplements recovery per engine and becomes the most dangerous code in the estate.
• RejectedNo PITR; frequent snapshots only: Cannot meet a 5-minute RPO or recover to just before a named event.

Consequences

What it buys

• Replay correctness is the engine's, tested by its community
• Explicit gaps make RPO honest
• Adding an engine means adding a range reporter

What it costs

• Granularity differs: to the second for PostgreSQL, to the event for MySQL, to the timestamp for TiDB
• Replay speed is whatever the engine gives, which is why ADR-19 exists

Choose differently when

Own the log only when many engines without native PITR need a minutes-scale RPO and restore into an environment the engines' tools cannot reach.

Why it holds up over time

Engines keep improving their own recovery, and this design benefits from that automatically. The range map is a plain data structure that any future engine can report into.

How do we stop version drift from producing copies that write cleanly and do not restore?

Context

Version drift is the most common cause of a backup that succeeds and cannot be restored: XtraBackup behind a MySQL minor release, pgBackRest meeting a new WAL format, a BR version incompatible with its cluster. Nothing fails at capture time. It fails months later at restore.

Decision

A nightly restore matrix certifies every engine version seen in the estate against every tool version in use at depth 3. Passing pairs are written to a signed file in Git. Every adapter checks its pair against the file before capturing and refuses if it is absent, which raises RPO-at-risk and pages the owner. Engine versions come from discovery, not declarations.

How it works on-premise

GitLab CI triggers the matrix when Renovate proposes a tool upgrade or discovery reports a new engine version. The matrix runs in the enclave with synthetic datasets per engine and signs the result with cosign. Adapters verify the signature and read the pair list at start, and each capture reads the live engine version.

Options weighed

• ChosenSigned certification gate, enforced at capture: Drift becomes a same-day blocked capture instead of a failure found at restore.
• RejectedPin versions by policy and trust teams: The versions that break things are the unplanned ones.
• RejectedRely on the daily rehearsal to catch drift: Tier 2 and 3 rehearse too rarely, and every copy in between may be bad.
• Right elsewhereWarn but capture anyway: Acceptable for Tier 3, where a blocked capture might cost more than a doubtful copy. Here it is not used even there, to keep one rule.

Consequences

What it buys

• No uncertified copy is written
• Upgrades are proven by a restore before they matter
• The certification list documents what works with what

What it costs

• An early engine upgrade blocks captures until certification, up to a day
• Matrix runs use enclave capacity overnight
• Synthetic datasets must cover each engine's features in use

Choose differently when

Relax to warn-and-capture for a datastore whose owner accepts, in writing, that a doubtful copy is better than none during an upgrade window.

Why it holds up over time

Engines and tools will keep releasing, faster rather than slower. A gate driven by what is actually running scales with that pace and needs no one to remember which versions are compatible.

How do we protect 1.3 PB of object storage without listing billions of objects every night?

Context

Tier 3 buckets change about 3% a day. A nightly listing to find changes costs time and gateway load in proportion to the total object count, not the change. Notifications are efficient but can be lost, and a missed notification is a silently unprotected object.

Decision

Source buckets have versioning and persistent RGW bucket notifications to Kafka. The object mirror consumes them, copies each new version to custody with an envelope, and records the event. A weekly reconciliation lists each bucket and compares it with the catalogue, and any difference is copied and reported as a finding.

How it works on-premise

Persistent notifications (queued by RGW until delivered) go to a three-broker Kafka topic per custody destination. The mirror is a Go consumer group that streams GET to PUT without staging and encrypts per object with the bucket's data key. Reconciliation shards listings by prefix and runs at low priority in the maintenance window.

Options weighed

• ChosenNotifications with weekly reconciliation: Work in proportion to change, plus a regular check that nothing was missed.
• RejectedNightly rclone listing sync: Listing cost grows with total objects and competes with Tier 1 for gateways.
• RejectedStorage-level replication to custody: Replicates deletes and corruption, and joins the domains.

Consequences

What it buys

• Capture load follows change
• Per-object envelopes allow per-object restore
• Reconciliation limits the damage a lost notification can do to one week

What it costs

• Kafka becomes a dependency for Tier 3 RPO
• Versioning on source buckets adds source storage
• Many small artefacts in the catalogue

Choose differently when

For buckets of large, rarely changing objects, a periodic listing is simpler and the notification pipeline is not worth running.

Why it holds up over time

Change feeds plus periodic reconciliation is a general pattern for keeping two stores in step, used well beyond backup. Either part can change (a different broker, a different listing tool) without changing the other.

How are several datastores restored to a mutually consistent point when their engines share no common quiesce?

Context

Two applications write to more than one datastore in one business operation (orders to PostgreSQL and TiDB, for example). Restoring each to the same wall-clock second does not guarantee consistency, because clocks differ and log positions do not line up. Coordinated freezes across engines would stall production.

Decision

A consistency group writes a marker row with the same group id to each member datastore every 5 minutes through the application's own connection pool. Group restore recovers each member to the log position containing a chosen marker. Each group must declare whether operations in flight between markers are safe to replay, and supply a reconciliation check that the rehearsal runs.

How it works on-premise

A small marker job per group, with credentials from OpenBao. Adapters index marker positions from WAL, binlog and TiDB log ranges into recovery_range. The group restore workflow restores members in parallel, each to the marker position, then runs the group's reconciliation assertion at depth 3.

Options weighed

• ChosenMarker transactions with a tested reconciliation: Consistency is approximate but aligned, and the remaining skew is checked, not assumed.
• RejectedCoordinated freeze across engines: Stalls production writes and has no cross-engine primitive.
• RejectedSame-timestamp restore with no alignment: Clock skew and commit ordering make it silently inconsistent.
• RejectedGroups for every multi-store service: Most services tolerate skew. Groups only for the two that cannot.

Consequences

What it buys

• Group restore to an aligned point without stopping writes
• Reconciliation is exercised by rehearsal
• Cost limited to two applications

What it costs

• Marker rows in application schemas
• Owners must write reconciliation checks
• Five-minute alignment granularity

Choose differently when

If an application moves to a single distributed database for both stores, the group becomes unnecessary and should be dissolved.

Why it holds up over time

Markers written into each store's own log survive any change of backup tool, because they are data, not tool metadata. Multi-store consistency is an application property and stays with the application.

If the catalogue is lost, are the copies still usable?

Context

A catalogue that is the source of truth is fast and queryable, and it is a single point of failure for data it does not hold. At 30 million artefacts a scan is slow, but not prohibitively so. A rebuild that has never been run is exactly the kind of unproven claim this platform exists to eliminate.

Decision

Every artefact has a signed envelope beside it in custody: engine and version, tool and version, consistency level, key id, chain parent, log range and checksum. The catalogue is derived. A full rebuild by scan runs every week into a shadow database and is diffed against the live catalogue. Restore can proceed from envelopes alone.

How it works on-premise

Envelopes are JSON with a schema version, signed with the adapter's SPIFFE identity and written under the same Object Lock. Keys are prefixed by datastore, day and artefact ULID, so a scan can be split. The weekly job uses 32 listers and a projection builder writing to a standalone PostgreSQL instance, and a diff mismatch is a finding.

Options weighed

• ChosenSelf-describing artefacts with a weekly rebuild: The index is an accelerator, and the rebuild is proven every week.
• RejectedCatalogue as the only source of truth, backed up: The catalogue's backup becomes a special case with its own unproven restore.
• RejectedNo catalogue; always scan: Finding a recovery point would take hours, which breaks the RTO.

Consequences

What it buys

• Catalogue loss delays reports but does not prevent restores
• Tools of the future can read today's copies
• Weekly diff catches catalogue bugs

What it costs

• Envelope writing is part of every adapter
• A few hours of custody listing load every week
• Orphan artefacts without envelopes must be handled

Choose differently when

If artefact counts reach billions, a full weekly scan stops being feasible. Move to incremental rebuild by day prefix and a monthly full scan.

Why it holds up over time

A copy that carries its own description stays usable when the catalogue software, the schema and even the organisation that built them have all changed. The envelope schema is the one format here designed to last a decade, and it is versioned for that reason.

How is catalogue state stored so that its history survives corrections?

Context

Artefact state changes: captured, copied, verified, held, expired. A mutable row loses what was claimed before a correction, which matters when auditing why a datastore was shown as PROVEN. The query load is modest: recovery-point lookups at p95 of 500 ms over 30 million artefacts.

Decision

catalogue_event is the only table written to. Status, recovery ranges, custody copies and proofs are projections updated in the same transaction and can be dropped and replayed. PostgreSQL 17 with Patroni, a synchronous replica in DC-A and a standby cluster streaming to DC-B.

How it works on-premise

Events are partitioned by month and never updated or deleted during retention. Projection tables are indexed on (datastore_id, from_ts) and (datastore_id, state). The catalogue itself is backed up with pgBackRest to custody, and is also independently rebuildable from envelopes (ADR-15).

Options weighed

• ChosenPostgreSQL event table with projections: Familiar, transactional and more than sufficient at this scale.
• RejectedKafka as the event log with stream-processed views: More moving parts for a workload with modest write rates.
• RejectedMutable relational rows: Loses the history of claims.
• Right elsewhereClickHouse: Good for analytics over proofs later. Wrong for transactional projections.

Consequences

What it buys

• Every past claim is reconstructable
• Projections can be fixed by replay
• Ordinary operational skills apply

What it costs

• Event growth must be partitioned and managed
• Projection logic must be idempotent
• Two recovery paths for the catalogue to keep healthy

Choose differently when

Move to a distributed log only if many independent consumers need the event stream in real time, not just the catalogue.

Why it holds up over time

An append-only log plus derived views has worked for decades, from accounting ledgers to database write-ahead logs. PostgreSQL has long-term community governance and an unusually conservative approach to compatibility.

Where do audit events and drill records live, and what happens if they cannot be written?

Context

Every capture, restore, retention change, hold, disposal and access to a restored copy must be an attributed, immutable event written before the action completes. Drill records underpin every RTO claim and must outlive the artefacts they verified. If evidence shares the catalogue's database, it also shares its administrators and its failures.

Decision

Destructive actions write an intent event first, and if that write fails the action does not start. Events are also sealed into hash-chained daily segments in an evidence bucket in custody B with compliance-mode Object Lock for 7 years. Verification and drill records follow the same path.

How it works on-premise

The API writes intent to catalogue_event and to an outbox in the same transaction. A sealer service writes segments hourly and closes each day with a hash that includes the previous day's hash, and publishes the daily hash to the Git repository as an external anchor. Auditors verify the chain with a small open-source CLI.

Options weighed

• ChosenIntent first, sealed segments under compliance lock: No record, no action, and evidence nobody can change.
• RejectedAudit table in the catalogue only: Catalogue administrators could alter it and its loss takes the evidence too.
• Right elsewhereLog shipping to a SIEM only: Useful for detection, and events are also sent there. It is not the evidence store.

Consequences

What it buys

• Tamper-evident history with an external anchor
• Evidence survives catalogue loss
• Auditors verify independently

What it costs

• An evidence outage blocks destructive actions
• A seven-year compliance lock cannot be undone if something is written by mistake
• One more service

Choose differently when

If a jurisdiction requires personal data in audit events to be erasable, store pseudonymous ids in sealed segments and keep the identity mapping in a separately erasable table.

Why it holds up over time

Hash chains with an external anchor are a long-established method, and their verification needs only the data and a hash function. Seven-year evidence remains checkable long after this platform's software has been replaced.

Who may restore, into what, and how is overwriting production made safe?

Context

Self-service restore brings RTO down and makes drills routine. Overwriting a production datastore with an older copy is the most destructive operation the platform can perform, and most recoveries do not need it: restoring side by side and copying back damaged rows keeps today's good writes.

Decision

Owners may self-serve side-by-side, partial and isolated restores into targets whose classification is at least as strict as the source. In-place restore requires an approver from the owner group who is not the requester, using WebAuthn, an intent record, and a pre-restore snapshot of every target volume kept 7 days. A failed snapshot stops the workflow.

How it works on-premise

OPA evaluates team ownership and target classification from NetBox and namespace labels. Keycloak provides WebAuthn step-up for approvals. The in-place workflow fences writers, takes Ceph RBD snapshots through the CSI snapshot API, restores, runs depth-2 checks and promotes. Reverting rolls back the snapshot. Bare-metal targets take a pgBackRest full copy instead.

Options weighed

• ChosenSelf-service side by side, gated in-place: Fast for the common case, and deliberate and reversible for the dangerous one.
• RejectedCentral team performs all restores by ticket: Slow restores, and drills become rare.
• RejectedFull self-service including in-place: One mistaken click overwrites today's data with last month's.

Consequences

What it buys

• Most restores need no second person
• In-place restores are reversible for 7 days
• Classification enforcement is automatic

What it costs

• Approver availability adds minutes to in-place restores
• Snapshots need capacity on production storage
• Owner metadata must be accurate for authorisation to work

Choose differently when

Restrict self-service to a central team where target classification cannot be enforced automatically, such as unlabelled legacy environments.

Why it holds up over time

Default to non-destructive and make the destructive path reversible: this holds however restores are implemented. The rules live in OPA policy, so they can change as the organisation changes without touching workflow code.

What actually determines whether a Tier 1 restore finishes in 60 minutes?

Context

Restore time is set by the slowest link. Transfer at 4 GB/s moves 4 TB in about 17 minutes, but PostgreSQL replays WAL on a single process. A day of WAL on a busy store can take over an hour to replay by itself. A base copy taken nightly makes the 60-minute RTO impossible for most of the day, whatever the storage throughput.

Decision

The Tier 1 RTO is broken into a budget: 10 min to authorise and start, about 17 min to transfer, at most 10 min of replay, and 10 min of assertions and promotion. Hourly block-incremental bases bound replay distance. The 60-minute RTO applies to recovery points in the last 7 days, while points from day 8 to day 35 have a stated RTO of 3 hours. Every drill records all four parts separately.

How it works on-premise

pgBackRest block incremental with bundling, run hourly and cheap because only changed blocks are copied. Weekly full, daily differential and hourly incremental, with chain depth up to 25 and at most 7 days from a full. Restores use process-max 32 across RGW gateways. Drill telemetry stores transfer, replay and assertion durations per run.

Options weighed

• ChosenBudgeted RTO with hourly bases: Makes the replay term, the usual limit, small and measured.
• RejectedNightly base with WAL: Replay alone can exceed the RTO late in the day.
• Right elsewhereWarm standby restored continuously: The lowest RTO, at the cost of a second copy of every Tier 1 store. It is also close to being a replica, which cannot replace a backup.

Consequences

What it buys

• The RTO holds throughout the day, not only after the nightly copy
• The four-part breakdown tells you what to fix when a drill is slow
• Owners see the RTO step change at day 7

What it costs

• Hourly capture load on Tier 1 sources
• Longer backup chains
• Two stated RTOs to explain

Choose differently when

If replay speed improves enough (parallel recovery in future engine versions), hourly bases can become daily. The drill breakdown will show when that point is reached.

Why it holds up over time

Whatever the technology, RTO is transfer plus replay plus checking. Measuring each part every day means the budget changes when the numbers change, instead of being inherited from a document.

How do we recover to before a corruption whose start time nobody knows?

Context

Logical corruption is usually found weeks after it started. An SRE guessing recovery points one at a time spends a multi-hour restore per guess. The platform can restore any second, but only the owner can say what 'clean' means.

Decision

Clean-point search takes an owner-supplied invariant and a window, restores 8 evenly spaced points in parallel in the enclave, evaluates the invariant, and narrows to the interval where it changes from clean to dirty. Seven rounds get from 35 days to one second. Every round is recorded as evidence. Tier 2 rehearsals pause during a search, and Tier 1 rehearsals continue.

How it works on-premise

A Temporal workflow fans out 8 restores through the normal RestoreWorkflow, each to a point in the current interval, then runs the invariant through the assertion runner and chooses the next interval. Round duration is dominated by replay distance, which the hourly bases (ADR-19) keep short within 7 days.

Options weighed

• ChosenParallel k-ary search with owner invariant: Turns days of guessing into hours of automated work with a record.
• RejectedManual binary search by SRE: Sequential, slow and unrecorded.
• RejectedReplay change log row by row looking for the bad write: Needs application semantics the platform does not have.

Consequences

What it buys

• Recovery point found in about 3 hours inside the hourly window
• Evidence of how the point was chosen
• Reuses rehearsal capacity and code

What it costs

• Only as good as the invariant
• Assumes corruption, once present, persists
• Consumes 8 enclave slots for the duration

Choose differently when

When corruption is intermittent or repaired and recurring, use the round results as input for a person rather than trusting the search's final answer.

Why it holds up over time

Searching a monotonic predicate over time is a general algorithm. Faster restores make each round cheaper, and more slots make it wider.

Can a restore be done when the platform itself is part of the outage?

Context

The day a restore is needed may be the day the control cluster, catalogue and console are down too, for example after a ransomware incident or a data-centre loss. A restore that needs the platform to be healthy has a hidden dependency on everything the platform depends on.

Decision

A written runbook restores any engine class using only a break-glass custody read role, the envelope next to the artefact, OpenBao to unwrap the key, and the engine's tool at the version recorded in the envelope. It is rehearsed every quarter with the control cluster switched off, by someone who did not write it, and timed.

How it works on-premise

Break-glass read credentials come from the sealed custody OpenBao after the usual two-officer unseal, from a read-only role kept separate from the bypass role. Tool container images at every certified version are mirrored to a registry at the vault site. A static binary of the bnr CLI lists envelopes by prefix and prints the exact restore command.

Options weighed

• ChosenDocumented, rehearsed, minimal-dependency runbook: The restore does not depend on the platform being healthy.
• RejectedHighly available control plane as the answer: Reduces the probability of the platform being down but not its correlation with the incident.
• RejectedRunbook written but not rehearsed: An unrehearsed runbook is another unproven claim.

Consequences

What it buys

• Restore survives platform loss
• The runbook's accuracy is proven quarterly
• Engineers learn the underlying tools

What it costs

• Quarterly gameday time
• The image mirror and CLI binary must be kept current
• Break-glass read access is one more credential path to guard

Choose differently when

Nothing flips this for Tier 1. For Tier 3, a manual path that is documented but only rehearsed yearly may be acceptable.

Why it holds up over time

Because the dependencies are minimal and are the standard building blocks (an S3 API, a JSON file, a key manager, a tool), the runbook stays short when the platform gets complicated. It also becomes the migration path if the platform is ever replaced.

What orchestrates captures, restores and rehearsals, and what happens when it is down?

Context

Restores and rehearsals are long, multi-step, wait on humans, retry throttled reads and must survive worker restarts. Cron can start a job but cannot resume a half-finished restore. Any orchestrator is also a dependency that can fail at the worst moment.

Decision

Temporal orchestrates captures, restores, rehearsals and searches. Tier 1 log shipping runs outside it in the engines' own processes. Base copy schedules are also rendered as suspended Kubernetes CronJobs beside agents, and a watchdog unsuspends them when Temporal's heartbeat is missing for two intervals. Degradation order is reporting, console, Tier 3 capture, verification. Tier 1 capture and restore never degrade.

How it works on-premise

Temporal server on the control cluster with PostgreSQL persistence (3 history, 2 frontend, 2 matching). Workers are Go. The policy compiler writes both Temporal Schedules and the CronJob manifests from the same compiled policy. The watchdog is a small DaemonSet on agent nodes. Envelopes written during fail-static mode are ingested when Temporal returns.

Options weighed

• ChosenTemporal with fail-static fallback: Durable, resumable workflows, without making capture depend on them.
• Right elsewhereArgo Workflows: A good fit for batch pipelines. Weaker for long human-in-the-loop waits and signals.
• RejectedKubernetes CronJobs only: No resumable restores, approvals or in-flight ETA escalation.
• RejectedCustom state machine in the API: Reimplements durable execution, badly.

Consequences

What it buys

• Restores survive worker crashes
• Approvals are signals, not polling
• Control-plane loss delays but does not lose Tier 1

What it costs

• Temporal is a significant system to run
• Two schedule representations to keep in step
• Workflow code versioning discipline

Choose differently when

For an estate of a dozen datastores with no human approvals, CronJobs and a simple restore script are proportionate.

Why it holds up over time

Durable execution is a well-established category with several implementations. Workflows are ordinary Go code calling adapters, so a move to another engine rewrites orchestration, not adapters or custody.

How does a new datastore get protected without someone remembering to set it up?

Context

Per-resource job configuration drifts and is forgotten for new resources. A default tier silently assigns an RPO nobody agreed. The biggest loss class is the datastore nobody enrolled, which a platform watching only its own jobs cannot see.

Decision

Protection policy binds a resource class and a declared tier. Owners declare the tier by label in Git, the NetBox custom field or the namespace. A resource with no tier is refused where admission allows (Kyverno for PVCs and operator CRs) and reported everywhere else. Discovery runs every 15 minutes and raises an incident for anything unprotected for more than 72 hours.

How it works on-premise

Policies are CRDs in Git, synced by Argo CD and compiled into schedules, retention and rehearsal slots. The coverage scanner reads NetBox, the Kubernetes API (PVCs, CloudNativePG and TiDB operator CRs), the RGW admin listing and OpenTofu state, and publishes unprotected age as a metric. Kyverno rejects a PVC without backup.tier in namespaces marked for production.

Options weighed

• ChosenTier-bound policy, no default, continuous discovery: New resources inherit protection, and unknown ones become visible.
• RejectedPer-datastore jobs configured in a UI: Drifts, and misses whatever nobody configured.
• RejectedDefault everything to Tier 2: An RPO and a cost that nobody agreed to.

Consequences

What it buys

• Protection is reviewed like code
• Unprotected resources have an age and an owner
• New services get protection without asking

What it costs

• Admission refusals annoy teams at first
• Inventory quality limits discovery
• Resources outside NetBox and Kubernetes need other sources

Choose differently when

Allow a temporary default with a short expiry during a large migration, when refusing every unlabelled resource would stop the migration.

Why it holds up over time

Policy on classes rather than instances scales with the estate and survives inventory tools changing. Discovery sources are pluggable, and the rule that unprotected is a reported state, not an unknown one, stays the same.

Where does everything run so that one site's loss leaves both copies and the ability to restore them?

Context

The platform must restore after losing a data centre, with its control plane possibly in that site. Backup and restore traffic runs at 3 to 4 GB/s and must not compete with production. The key service must keep quorum through a site loss.

Decision

Custody A in DC-A, custody B and the enclave in DC-B, tape and one OpenBao voter at the vault site. The control plane is active in DC-A and warm in DC-B, with a Patroni standby catalogue and the Argo CD application defined but scaled to zero. Backup traffic runs on a dedicated 2 × 100 GbE network. A regional loss of both data centres is accepted as a tape-only recovery.

How it works on-premise

OpenBao voters: 2 in DC-A, 2 in DC-B, 1 at the vault site. The Patroni standby cluster streams to DC-B. Starting the DC-B control plane is a runbook step (promote the standby, scale up) timed at gamedays. Capture and gateway hosts have dual 100 GbE ports on a separate spine from the service network.

Options weighed

• ChosenTwo data centres plus vault site, warm control: Survives a site loss with both a copy and a way to restore it, at modest standing cost.
• RejectedActive-active control plane: Adds Temporal multi-cluster complexity for a plane whose loss delays but does not lose data.
• DeferredThird data centre with a custody cluster: Priced. Revisit if the regional risk is not accepted by the board.
• RejectedBackup traffic on the production network: 3 GB/s of backup traffic becomes a production incident.

Consequences

What it buys

• Site loss survivable with measured RTO
• Key service keeps quorum
• No backup traffic on production links

What it costs

• Warm standby must be exercised to be real
• Dedicated network hardware
• Regional risk accepted, not removed

Choose differently when

Add a third custody site when the regulatory set grows or the regional risk is no longer accepted. The copier pattern (ADR-08) extends to a third target without redesign.

Why it holds up over time

Physical failure domains, independent networks and quorum across three sites are infrastructure basics that have not changed in decades. Each piece can be refreshed on its own hardware cycle without changing the topology.

Who can read a copy, and how do we make a restore into the wrong environment fail?

Context

Custody administrators must never be able to read plaintext. Restores into a less protected environment must fail by construction. Lawful erasure must be possible across immutable copies. And a copy whose key is lost is not a backup.

Decision

Adapters encrypt before bytes leave the source with a data key, per datastore per month for the regulatory set and per datastore per week otherwise. Data keys are wrapped by a key-encryption key per data classification in OpenBao Transit. Restore and verify identities may unwrap only for classifications their target is cleared for. Key versions are kept for longer than the longest retention and backed up offline under two-of-five escrow.

How it works on-premise

OpenBao runs with 5 Raft voters across DC-A, DC-B and the vault site. Tools use their native encryption (pgBackRest repository cipher, xbstream encryption, BR crypter, ClickHouse encrypted disk, Kopia repository key), keyed by the unwrapped data key. The envelope records the key id, version and algorithm. Transit exports are wrapped to an offline key held in escrow, and restoring them is drilled every quarter.

Options weighed

• ChosenEnvelope encryption, per-classification wrapping keys: Cryptographic separation of environments, cheap rotation, and erasure by destroying a key.
• RejectedServer-side encryption in custody: Custody administrators could read plaintext, and a restore could go anywhere.
• RejectedOne key per tool configured in files: No classification separation, and rotation means re-encrypting data.
• Right elsewhereHSM-backed key hierarchy: Adds tamper resistance where the regulator requires it, and can sit under OpenBao as a seal.

Consequences

What it buys

• A wrong-environment restore fails at unwrap
• Rotation rewraps kilobytes, not petabytes
• Erasure by destroying a key covers tape too

What it costs

• OpenBao is on the critical path for capture and restore
• Tool-native encryption differs per engine and has to be certified
• Offline key escrow has to be managed and drilled

Choose differently when

If every engine and tool supported a common external key management protocol natively, move wrapping into that protocol and remove per-tool configuration.

Why it holds up over time

Envelope encryption lets algorithms and key managers change without touching stored data: rewrap under a new key or a post-quantum wrapping algorithm, and the petabytes stay as they are. The envelope's key id and algorithm fields exist for that move.

How are the four identities in the requirement made real, and what is stolen if a host is compromised?

Context

The requirement separates capture, custody administration, restore and verification with disjoint rights. Static S3 keys and database passwords on capture hosts are the credentials most likely to be stolen, and they last until someone notices.

Decision

Custody has four RGW STS roles: capture (put and get, no delete, no retention change), custody admin (lock configuration, no object read), restore (get and extend retention) and verify (get only, from the enclave proxy). Workloads assume roles with SPIRE JWT-SVIDs. Source database credentials are issued per run by OpenBao's database engine. No long-lived credential exists in the platform.

How it works on-premise

SPIRE server per data centre with the Kubernetes and node attestors. RGW is configured with SPIRE's OIDC discovery endpoint as a web-identity provider, and each role's trust policy matches a SPIFFE id path. STS sessions last 1 hour. OpenBao issues PostgreSQL and MySQL users with a 2-hour lease and a grant limited to backup functions.

Options weighed

• ChosenSTS roles via workload identity: Nothing long-lived to steal, and rights per purpose.
• RejectedStatic RGW keys in Kubernetes Secrets: Long-lived and copyable, and often over-privileged.
• RejectedOne service account for the platform: A single compromise has every right, including the ones that must be separated.

Consequences

What it buys

• A stolen capture identity can write but never destroy
• Credentials expire within hours
• Rights are auditable per role

What it costs

• SPIRE becomes a trust root for custody roles
• RGW STS web identity must be proven in the PoC
• Database engines need backup-scoped roles designed per engine

Choose differently when

Where SPIRE is not available, Kubernetes projected service-account tokens can be the web identity, at the cost of weaker node attestation.

Why it holds up over time

Short-lived credentials from attested identity is where the whole industry is heading, and SPIFFE is a CNCF graduated standard. Roles defined by purpose remain valid whatever issues the tokens.