Architecture Decision Record

Secrets Distribution Service · Solution Architecture v1.0 · Security Platform Architecture · 2026-09 · 34 views · open source, fully on-premises

The argument these decisions serve is summarised in the Architecture One-Pager.

Thirty-four decisions make up this architecture. Everything else on the thirty-four views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still hold after the products, the team and the estate have changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Latencies, volumes and costs are the requirement's targets or planning estimates, and each is replaced by a measurement in the proof phase. Seven product behaviours are relied on and must be confirmed on the exact releases and hardware chosen before production depends on them: OpenBao's PKCS#11 seal working against the chosen HSM model, including the loss of one HSM of the pair; a single OpenBao active node sustaining 2,000 lease writes a second at commit p99 ≤ 50 ms on NVMe, and restoring expiration state for 90,000 leases after a leader election inside the 10-minute RTO; OpenBao's JWT auth method validating SPIRE JWT-SVIDs against a refreshed key set within the five-minute rollover; SPIRE's Delegated Identity API serving SVIDs for 110 pods on one node, with the Kubernetes workload attestor producing image-digest selectors on the container runtime in use; PostgreSQL 16 CREATEROLE scoping the broker's admin role to roles it created, and revocation that terminates sessions and drops roles inside 30 s at p99 under load; Envoy's credential injector taking a rotated credential over SDS without failing in-flight requests; and Kafka acks=all producer latency at p99 ≤ 10 ms in-site for the synchronous audit class.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it works on-premise: The concrete mechanism: which package, configured how, on whose hardware.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

Trust and identity: Where the chain of trust ends, how a workload becomes known, what it holds between requests, and how environments are kept apart without trusting policy to do it.

• ADR-01 · The chain of trust ends at attestation by the infrastructure, through SPIFFE and SPIRE
• ADR-02 · Identities are registered from a reviewed registry, and image digests come from the delivery pipeline
• ADR-03 · Sessions are short, bound to the presenting key, and carry the partition's restore generation
• ADR-04 · One trust domain per environment, so cross-environment access fails in the handshake

Partitions and placement: What one compromise or one failure can reach, where the platform itself runs, and what crosses between sites.

• ADR-05 · Partition by environment, site and shape: KMS, custody and broker are separate clusters
• ADR-06 · Tier 0 runs on dedicated enclave hosts under systemd, with nothing it serves underneath it
• ADR-07 · Two sites and nothing quorums across them; custodial versions are written to both before they are served

Build and adopt: Which engine to adopt, and the small set of checks that have to be built in front of it.

• ADR-08 · Adopt OpenBao as the broker and custody engine; build only what it cannot enforce
• ADR-09 · A stateless issuance gateway sits in front of the engines

Custody and keys: Who holds the root key, what a custody store is worth to an attacker, why custody and brokerage are different products, and how a key is used without being handed out.

• ADR-10 · The KMS is a separate OpenBao cluster on an HSM seal, run by key custodians
• ADR-11 · Custodial material is envelope-encrypted per version and unwrapped on every read
• ADR-12 · Brokerage and custody are two products with two guarantees, declared per secret
• ADR-13 · Non-exportable secrets are transit keys; the platform returns operations, never material

Brokered issuance: How long credentials live, what ends them when nobody is watching, the credential that creates credentials, upstreams that cannot revoke, and protecting upstreams from the platform.

• ADR-14 · Lease lifetimes: the platform sets bounds, roles set values, callers may only shorten
• ADR-15 · Leases hang off sessions, so a vanished workload's credentials expire with its session
• ADR-16 · Upstream admin credentials are scoped to one upstream in one environment, rotated daily and never readable
• ADR-17 · Irrevocable upstreams are brokered with 15-minute unrenewable leases and registered as risks
• ADR-18 · Per-role lease caps and per-upstream mint budgets shed issuance before an upstream is exhausted

Delivery: What runs next to the workload, how a credential reaches a process, what happens when the control plane is gone, and time.

• ADR-19 · One credential agent per node, built, using SPIRE's delegated identity and holding everything in memory
• ADR-20 · Three delivery modes, the socket first; environment variables are refused
• ADR-21 · Fail static until expiry; no grace extension exists anywhere
• ADR-22 · Agents age credentials on a monotonic clock and refuse to serve beyond ±30 s of skew

Rotation and revocation: Who owns the last mile of rotation, what contained means, how two systems of record are kept honest, and how a restore avoids reviving the past.

• ADR-23 · Rotation keeps two working credentials and retires on observed use, with a seven-day ceiling
• ADR-24 · Revocation by scope resolves to entries, sessions and leases, and reports contained only when the upstream confirms
• ADR-25 · Orphaned accounts are reconciled hourly against upstream account lists
• ADR-26 · A restore advances a generation held outside the restored store, and revalidates before serving

Governance, audit and people: How policy is made, how blast radius is known, how reads are recorded and kept, how a human reads a secret, and how leaks are noticed.

• ADR-27 · Policy is compiled from a reviewed registry; nobody writes policy
• ADR-28 · Blast radius is computed from compiled policy, per identity, per node and per shard
• ADR-29 · Audit posture is declared per event class: synchronous quorum for custody and humans, local-durable and shipped for renewals
• ADR-30 · Audit lands in ClickHouse for 400 days and in Ceph Object Lock for seven years, owned by the records team
• ADR-31 · No standing human read; break-glass is two-person, reasoned, response-wrapped and single use
• ADR-32 · Leak detection joins upstream logs to leases; decoys are real accounts with no privileges

Operations and cost: How the platform changes without becoming the outage, and how the cost of a security choice is put in front of the person making it.

• ADR-33 · Releases go partition by partition and site by site, gated by an invariant suite
• ADR-34 · Lifetime is published as load and cost, shown rather than charged, and cache hit rate is a headline metric

Technology by capability

Every capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns; HSMs are the only component that is not software, and upstreams are integrated but not operated.

The decisions, and the alternatives that lost

Trust and identity

Where the chain of trust ends, how a workload becomes known, what it holds between requests, and how environments are kept apart without trusting policy to do it.

ADR-01 · The chain of trust ends at attestation by the infrastructure, through SPIFFE and SPIRE

Status: Accepted · Shown on views: 10, 17, 30

What does a workload present to prove what it is, and where does that chain of proof finally end?

Context. Every secrets platform has to answer secret zero: the credential a workload uses to obtain its credentials. If the answer is a token injected by the pipeline or baked into an image, the platform has concentrated the estate's secrets behind one more valuable secret. The requirement asks for an attestation produced by the infrastructure, verified against the issuer's published keys, bound to cluster, namespace, service account and image digest, refused for unregistered coordinates, and a register for every place where no attestation is possible. On premises there is no cloud identity federation to lean on. What the organisation does own is a Kubernetes control plane that already signs service-account tokens and a kubelet that already knows what each container is.

Decision. Use SPIFFE as the identity contract and SPIRE as its implementation. Kubernetes nodes attest with the SPIRE agent's projected service-account token, verified by TokenReview against the cluster's API server. Pods are attested by the SPIRE agent through the kubelet on namespace, service account and image digest. The attestation presented to the platform is a five-minute JWT-SVID, sent over mTLS authenticated with the same pod's X.509-SVID. Bare-metal and VM hosts move to TPM DevID node attestation in Phase 3; until then a VM joins with a single-use join token that is a registered secret-zero exception.

How it works on-premise. Three SPIRE servers per environment per site on a Patroni-managed PostgreSQL datastore, each deployment holding a site intermediate signed yearly by the environment's offline root in an HSM. SPIRE agents run as a DaemonSet in every estate cluster. The API server service-account signing keys live on control-plane hosts with TPM-bound disk encryption and are registered as trust anchors, with their own custody and rotation procedure.

What it buys

• No image, repository, pipeline or configuration file holds anything a workload uses to authenticate
• Identity describes what is running, including its digest, rather than what someone declared
• One identity format that the service mesh, databases and future federation can also consume

What it costs

• The API server's signing key becomes a trust anchor that must be guarded like a CA key
• SPIRE becomes a Tier 0 dependency with its own datastore, upgrades and drills
• VMs carry registered exceptions until TPM attestation is in place

Choose differently when. If most workloads ran on VMs or bare metal rather than Kubernetes, TPM node attestation should be the MVP path and service-account tokens the addition. If the estate lived on a single managed cloud, the provider's workload identity federation is the better root and SPIRE is optional.

Why it holds up over time. A SPIFFE ID is a URI and an SVID is an X.509 certificate or a JWT; both are open standards consumed by Istio, Cilium, Envoy and others. A later change of implementation, or of attestation method from tokens to TPMs, changes how an SVID is obtained and not what it says, so grants, audit history and dashboards remain valid.

Lesson. Secret zero is not solved by hiding the first secret better. It is solved by asking something that already knows the answer and cannot be talked out of it.

ADR-02 · Identities are registered from a reviewed registry, and image digests come from the delivery pipeline

Status: Accepted · Shown on views: 04, 14, 16

How does a workload become known to the platform, and what stops an unknown one being trusted because it looks plausible?

Context. SPIRE's Kubernetes controller can create an identity for any pod that matches a template, which is identity on first sight: anyone able to create a pod in a matching namespace gets one. The requirement forbids that. It also asks that identity be bound to the image digest where the substrate supports it, but digests change on every build, so a registry of digests maintained by hand would be wrong within a day.

Decision. Identities are declared in the registry with environment, cluster, namespace, service account and owner. The compiler emits one exact SPIRE entry per identity and one OpenBao JWT role with an exact bound subject. The image-digest selector is written into the entry by the delivery pipeline at deploy time, from a signed image that admission control has accepted. An image not deployed by the pipeline has no matching entry, and its pod receives no identity.

How it works on-premise. The registry is a Forgejo repository of YAML. The compiler runs in CI and renders ClusterStaticEntry objects for spire-controller-manager, applied by Argo CD. The release job patches the digest selector for the workload it is deploying, using an identity allowed to change only that selector.

What it buys

• No identity exists that a person did not approve
• A hotfix pushed around the pipeline surfaces as a clear denial, not as a silent new identity
• Every identity has an owner who is paged when it is over-granted or orphaned

What it costs

• Every deploy writes to SPIRE, so the pipeline is on the path to a new version's first credential
• Images pushed by hand in an incident fail until run through the pipeline
• A slow registry review becomes a slow launch

Choose differently when. A small estate with a handful of teams and a strict admission controller can accept namespace and service-account templates, trading the registry's overhead for a boundary it can still reason about.

Why it holds up over time. The registry's schema names concepts (identity, secret, role, grant, owner), not SPIRE or OpenBao objects. If either product is replaced, the compiler changes and the approvals do not.

Lesson. If a new identity can appear without a person deciding it should, authentication is only as strong as the least-guarded deploy right.

ADR-03 · Sessions are short, bound to the presenting key, and carry the partition's restore generation

Status: Accepted · Shown on views: 12, 17, 24

What does a workload hold between attestation and each request, and what stops it being useful to anyone else?

Context. OpenBao tokens are bearer tokens. Address binding exists but is not key binding, and pods change addresses. The requirement asks for session tokens bound to the client's key, 15 minutes long and renewable to 12 hours, and for a store restore to invalidate every session issued before it. X.509-SVIDs rotate their keys every 30 minutes, so any binding must follow rotation without a new attestation.

Decision. The gateway wraps the engine's token into a session token: an authenticated-encryption envelope holding the OpenBao token, the SPIFFE ID, a SHA-256 thumbprint of the presenting SVID's public key, the partition generation and an expiry. On every request the gateway checks that the TLS client key matches the thumbprint and that the generation is current. The agent renews the session about every ten minutes over its current SVID, which rebinds the thumbprint to the new key.

How it works on-premise. The session key is an AES-256-GCM data key issued by the KMS, rotated daily and held only in gateway memory, with the previous key accepted for one rotation. Gateways read the partition generation from the KMS every ten seconds.

What it buys

• A session token found in a log, a dump or a packet capture cannot be used
• Restoring a store invalidates every earlier session in one operation
• The gateway holds no per-session state and scales by adding instances

What it costs

• A token format owned by the platform, which needs fuzzing, versioning and review
• The session key is a secret in gateway memory, derived from the KMS and rotated daily
• Renewal must stay ahead of SVID rotation, a timing contract the agent has to honour

Choose differently when. If the engine gained native certificate-bound tokens and a generation check, the wrapper should be deleted and the engine's token handed to the agent directly.

Why it holds up over time. The binding applies RFC 8705's idea of certificate-bound tokens at the gateway. The format is versioned and invisible to applications, and every session in existence expires within 12 hours, so a new format can replace the old one in a single release with no migration.

Lesson. A token that works from anywhere is a password with a timer. Bind it to a key its holder cannot hand over.

ADR-04 · One trust domain per environment, so cross-environment access fails in the handshake

Status: Accepted · Shown on views: 10, 12, 31

How is a non-production identity kept out of production secrets in a way that no policy mistake can undo?

Context. The requirement asks that cross-environment access be denied structurally, not by any policy a human can write. Policy languages can express anything, including a wildcard that grants staging a production path. The check must happen before policy is consulted and must fail closed when anyone gets it wrong.

Decision. Each environment is its own SPIFFE trust domain under its own offline root. Production gateways trust only the production bundle, so a staging SVID cannot complete a TLS handshake with them. Engines, KMS clusters and upstream admin credentials are per environment. No federation relationship exists between environment trust domains. As defence in depth, the gateway also compares the trust domain with the partition before routing.

How it works on-premise. Three offline roots in HSMs, one per environment, and separate SPIRE deployments. The gateway's client CA pool contains only its environment's bundle. An estate cluster belongs to exactly one environment; clusters that mix environments are not permitted to run the agent.

What it buys

• A policy error cannot reach across environments, because policy is never evaluated
• A staging compromise yields nothing usable in production
• Audit and blast-radius reporting partition naturally by environment

What it costs

• Three PKIs, three sets of ceremonies and three SPIRE deployments
• Mixed-environment clusters must be separated before they can adopt the platform
• Tools that span environments need one identity per environment

Choose differently when. An estate whose non-production and production workloads share clusters cannot adopt this until those clusters are split. In the meantime, a shared trust domain with the gateway prefix check is the interim, recorded as an exception.

Why it holds up over time. It rests on X.509 path validation, the most heavily scrutinised code in any TLS stack. It depends on no policy language, no engine and no product feature, and it stays correct however the platform above it changes.

Lesson. Put the boundary you can never afford to get wrong where getting it wrong fails closed and loudly: in a handshake, not in a rule.

Partitions and placement

What one compromise or one failure can reach, where the platform itself runs, and what crosses between sites.

ADR-05 · Partition by environment, site and shape: KMS, custody and broker are separate clusters

Status: Accepted · Shown on views: 09, 25, 32

What is the unit of compromise and failure, and how many copies of the platform does the organisation run?

Context. An OpenBao cluster holds one barrier keyring in memory on its active node; anyone who reads that memory reads everything the cluster stores. Namespaces separate policy, not memory. The requirement asks that one broker compromise not yield another partition's material, that upstream admin credentials never span environments, and that the compromise of storage, key manager or broker memory each yield a bounded and stated subset.

Decision. For each environment at each site, run one KMS cluster, one custody cluster, a set of broker shards divided by upstream family, one gateway pool and one SPIRE deployment. Brokers and custody never share a barrier. The partition map, which says which endpoint serves which environment, site and shape, is data the gateway and agents read.

How it works on-premise. Production at each site: three KMS voters with an HSM pair, five custody voters, two broker shards of five voters, six gateways, three SPIRE servers and a three-node datastore. Staging runs the same shape at both sites on VMs with three voters per cluster; development runs at site A only.

What it buys

• The blast-radius statement for each component is short and true
• Upgrades, drills and incidents are staged one partition at a time
• A storm of brokered writes cannot starve custodial reads

What it costs

• About thirty machines per production site, and many clusters to patch and drill
• Tooling must understand the partition map
• Some upstreams need a broker shard of their own to stay within a clean boundary

Choose differently when. An organisation with a few hundred workloads and a single environment that matters should run one cluster per environment and accept a shared barrier. The three-way split costs more than it contains at that size.

Why it holds up over time. Partitions are rows in a map, not code paths. Adding a shard, a site or a new trust tier such as a payment-card enclave is a new row and a new cluster of the same shape, not a redesign.

Lesson. Draw partition boundaries around what an attacker would get from one memory image, not around what is convenient to operate.

ADR-06 · Tier 0 runs on dedicated enclave hosts under systemd, with nothing it serves underneath it

Status: Accepted · Shown on views: 25, 26, 31

Where does the platform itself run, given that almost everything in the estate needs it in order to start?

Context. After a site-wide power event, databases, Kubernetes workloads and even the estate's CI need credentials before they can start. If the platform ran on the estate's Kubernetes, it would wait on a cluster whose workloads wait on it, and the estate's node administrators could read the platform's memory.

Decision. KMS, custody, broker, gateway and SPIRE servers, with their datastore, run on dedicated bare-metal hosts in a security enclave: their own racks in three rooms per site, their own VRF, static binaries under systemd, configured by Ansible from signed artefacts already staged on each host. The governance plane runs on a small separate Kubernetes cluster that is an ordinary consumer of the platform.

How it works on-premise. Disks encrypted with LUKS and bound to the TPM, Secure Boot, no interactive shell except through a recorded bastion, nftables allowing only the gateway port from the estate, and time from two independent stratum-1 sources through chrony.

What it buys

• A cold start of the whole site has a defined order, and the platform is first
• The set of people with host access to Tier 0 is small and named
• Failure analysis involves no orchestrator

What it costs

• Host management owned by the platform team rather than the estate platform
• No bin-packing; capacity for storms is provisioned in advance
• Slower to add capacity than a pod scale-out

Choose differently when. An organisation that already runs a hardened Tier 0 Kubernetes platform for identity systems, with a tested bootstrap and restricted administration, can reasonably run the engines there.

Why it holds up over time. The rule is testable and independent of technology: nothing Tier 0 needs at start may itself need Tier 0. A yearly cold-start drill checks it whatever runs underneath.

Lesson. The service everyone waits for at boot has to be able to boot before everyone.

ADR-07 · Two sites and nothing quorums across them; custodial versions are written to both before they are served

Status: Accepted · Shown on views: 13, 20, 25

How does the platform survive losing a site without letting one site's failure or compromise reach the other?

Context. The requirement asks for lease state per region, custodial material replicated between regions with an RPO of zero, and no cross-region call on the issuance path. OpenBao's community edition has no cross-cluster replication. A stretched cluster would put inter-site latency on every write and would need a third location to break ties.

Decision. Every cluster is local to its site. Leases and sessions are never replicated: a site outage is survived by the other site's workloads, not by failing leases over. Custodial versions are encrypted separately under each site's own envelope key, written to both custody clusters as staged, and promoted to active only when both writes have committed. Audit replicates asynchronously.

How it works on-premise. The rotation controller and the registry write path hold encrypt-only permission on both sites' KMS clusters. Neither site's KMS key material ever leaves its site. An emergency rotation during a site outage writes to the surviving site only, marks the version single-site, and the returning site is forced to that version before it reopens.

What it buys

• Sites fail, and are compromised, independently
• No witness location is needed
• Custody RPO is zero without a stretched quorum

What it costs

• Workloads must run at both sites to survive a site loss, which is an estate obligation
• Routine rotation pauses while either site is down
• Two ciphertexts per version to verify and reconcile

Choose differently when. An organisation with a metro pair under two milliseconds apart and a third location may prefer a stretched custody cluster, trading coupling for simpler writes.

Why it holds up over time. The rule that no quorum spans sites can be checked by reading the deployment inventory, and adding a third site is additive: one more write before promotion.

Lesson. Replicate what you cannot recreate and keep local what you can. A lease can be minted again; a lost key version cannot.

Build and adopt

Which engine to adopt, and the small set of checks that have to be built in front of it.

ADR-08 · Adopt OpenBao as the broker and custody engine; build only what it cannot enforce

Status: Accepted · proof required · Shown on views: 07, 09, 29

Build a secrets engine, adopt an open-source one, or buy one?

Context. The requirement needs a lease manager with revocation handles, dynamic credential plugins for 14 kinds of upstream, versioned key-value storage, non-exportable cryptographic operations, policy evaluated on every request and audit devices that block on failure. It must be open source and run on premises. HashiCorp Vault moved to the Business Source License in 2023 and no longer qualifies. OpenBao is the Linux Foundation fork of Vault's last open-source release, under MPL 2.0.

Decision. Adopt OpenBao for leases, dynamic credentials, KV v2, transit and policy evaluation. Build the issuance gateway, the credential agent, the registry compiler and applier, the rotation controller, the reconciler, the access broker, and plugins for upstreams that have none. Keep each built component small enough for one engineer to read in an afternoon.

How it works on-premise. A pinned OpenBao release on integrated Raft storage over NVMe. Plugins are built in CI and registered by checksum. Each cluster has one active node and standbys; broker clusters are sharded by upstream family to spread writes.

What it buys

• Leases, revocation, plugins and blocking audit devices are mature and understood
• Policy is evaluated per request against the current policy set
• The built code is limited to the requirements no engine enforces

What it costs

• One active writer per cluster, which forces sharding and a proven write ceiling
• A younger project community than Vault's, whose security release cadence must be watched
• Built plugins are tied to OpenBao's plugin interface

Choose differently when. If the proof shows a single shard cannot sustain 1,000 lease writes a second at the target latency, keep OpenBao for custody and transit and move lease state to a horizontally scalable store behind the gateway.

Why it holds up over time. No application sees OpenBao's API; the agent protocol is the contract. OpenBao can give way to a relicensed Vault, a successor fork or a built engine behind the gateway without a single workload changing.

Lesson. Adopt the engine and own the edges. The code that encodes your requirements should be small enough to read in an afternoon.

ADR-09 · A stateless issuance gateway sits in front of the engines

Status: Accepted · Shown on views: 12, 17, 22

Where do the checks live that the engine cannot express, and is another hop worth it?

Context. OpenBao cannot bind a token to a client key, evaluate environment before policy, enforce a restore generation, put renewals ahead of first issuance, apply a different audit posture to different event classes, or require a second identity to decrypt custodial material. Each is a stated requirement. Without a component that owns them they would be scattered across plugins, agent code and conventions.

Decision. Build a Go gateway, stateless and horizontally scaled. It terminates mTLS against the environment bundle, validates the JWT-SVID against the connection, wraps and checks sessions, enforces the generation, rate-limits per identity and per secret, runs two admission lanes, routes to engines by the partition map, unwraps custodial ciphertext through the KMS, and writes synchronous audit records for custodial and human events.

How it works on-premise. Six gateways per production site, twelve during a planned storm. Agents hold a static, signed endpoint list per partition and balance on the client side, so no load balancer and no DNS sits on the issuance path.

What it buys

• The engine can be replaced without re-implementing the requirements
• Capacity is added by adding instances, with no state to move
• One place to fuzz, review and test the security-critical checks

What it costs

• About two milliseconds of added latency
• Security-critical code owned by the platform team
• One more component to release and upgrade

Choose differently when. If the engine gains key-bound tokens and per-path audit posture, the gateway shrinks to rate limiting and admission and could become configuration on an off-the-shelf proxy.

Why it holds up over time. The gateway's responsibilities are a list derived from requirements, not from a product's permanent gaps. Each can be deleted on its own when an engine provides it, so the gateway gets smaller with time rather than accreting.

Lesson. Write down what the product cannot do, build exactly that, and keep the list so you can delete it later.

Custody and keys

Who holds the root key, what a custody store is worth to an attacker, why custody and brokerage are different products, and how a key is used without being handed out.

ADR-10 · The KMS is a separate OpenBao cluster on an HSM seal, run by key custodians

Status: Accepted · proof required · Shown on views: 09, 13, 25

What is the root of trust for material at rest, and who controls it?

Context. The requirement asks for a key manager the platform does not control, with independent access control, its own audit of every key use, and the property that withdrawing access leaves the platform unable to decrypt anything. There is no cloud KMS on premises. HSMs are the right place for a root key but make poor per-read envelope services: identity-aware policy and rich audit would have to be rebuilt around them.

Decision. For each environment at each site, run a three-node OpenBao cluster used only for transit keys and as the auto-unseal for that partition's custody and broker clusters. Seal it with an HSM pair over PKCS#11. The key custodians in the security office operate it; the platform team holds no policy on it. Its audit devices ship to the records team. Withdrawing the gateway's decrypt policy stops custodial reads on the next request; sealing the engines, a command custodians hold, stops everything else at once.

How it works on-premise. A pair of FIPS 140-3 Level 3 network HSMs per site. Recovery keys split three of five on hardware tokens held by named custodians. KMS hosts in their own racks. The engines authenticate to the KMS for auto-unseal with a narrowly scoped periodic token stored on TPM-bound encrypted disks; that token is the platform's own first secret-zero exception and is registered as such.

What it buys

• The key manager's log is independent of the platform's own
• Access withdrawal is real and exercised in drills
• Root key material never exists outside an HSM-sealed barrier

What it costs

• HSM hardware, support contracts and a custodian rota
• Another cluster per partition on the critical path of custodial reads
• An unseal token on engine hosts, written down as an exception rather than hidden

Choose differently when. If the organisation already runs an enterprise key management service with separate administration and its own audit, use its HSMs as the seal root and keep the OpenBao KMS layer for identity-aware envelope operations.

Why it holds up over time. Envelope keys are versioned. Moving to new HSMs, larger keys or a post-quantum algorithm is a key rotation followed by an in-place rewrap of ciphertext, which never exposes plaintext to storage and never touches a workload.

Lesson. Separation of duty is a property of who can use a key, not of which server the key sits on.

ADR-11 · Custodial material is envelope-encrypted per version and unwrapped on every read

Status: Accepted · Shown on views: 13, 18, 32

What does possession of the custody store yield, and what should a KMS outage do to custodial reads?

Context. OpenBao's barrier encrypts storage, but custody nodes hold the decrypted barrier key in memory, so reading one node's memory reads every secret. The requirement asks for every key use to appear in the key manager's own log, for custodial reads to fail closed when the key manager is down, and for storage, key manager and engine memory each to yield a bounded subset when compromised.

Decision. Custody stores only ciphertext produced by the KMS under a per-site envelope key. The workload's policy authorises reading the ciphertext; only the gateway's identity may decrypt it. Decryption happens on every read, and no decrypted copy is cached anywhere in the enclave. Ciphertext is bound to its secret path with a derived-key context, so ciphertext copied from one secret to another will not decrypt.

How it works on-premise. Transit keys with derivation enabled and the secret path as context. One envelope key per environment per site, rotated yearly with lazy rewrap on read and a background rewrap job. Versions are capped at 64 KB.

What it buys

• Storage, custody and KMS compromises each yield a stated, smaller subset
• The KMS log is a complete read history the platform team cannot edit
• Ciphertext cannot be swapped between secrets

What it costs

• The KMS sits on the custodial read path, adding two to five milliseconds and an availability dependency by design
• The KMS must be sized for custodial read volume
• Renaming a secret requires a rewrap

Choose differently when. If custodial reads grew into thousands a second, cache per-version data keys, never plaintext, in gateway memory for a few minutes, and accept a coarser KMS log.

Why it holds up over time. Envelope encryption with an external key manager is the construction every commercial KMS implements. Nothing about it depends on OpenBao, and any future custody engine can store the same ciphertext.

Lesson. If one component holds both the ciphertext and the key, you have moved the vault rather than built one.

ADR-12 · Brokerage and custody are two products with two guarantees, declared per secret

Status: Accepted · Shown on views: 02, 13, 20

Broker every credential and refuse to store anything, or accept a custodial tier, and how are the two kept from being confused?

Context. Partner API keys, signing keys and legacy accounts cannot be minted, and they are often the most consequential secrets in the estate. A platform that refuses them does not make them go away; they go back into CI variables. A platform that accepts them reintroduces long-lived material and a store worth stealing. The danger in having both is that people assume the custodial secret expires like the brokered one.

Decision. Accept a custodial tier as an admitted compromise with its own guarantee. Each secret's shape is declared in the registry and never inferred. Brokered roles promise a lifetime and revocation. Custodial secrets promise custody, controlled release, rotation and a maximum age. They live in different clusters, under different paths, with separate dashboards and SLOs. The agent presents one interface, so a secret that moves from custody to brokered changes no application code. The inventory marks each custodial secret whose upstream could support minting, as a migration candidate.

How it works on-premise. Brokered roles under database/, rabbitmq/ and built plugin mounts on broker shards; custodial secrets under kv/ and transit/ on custody clusters. The registry schema makes shape a required field and rejects a custodial secret without an owner, a rotation posture, a maximum age and a consumer set.

What it buys

• Nobody is told a partner key expires when it does not
• Each tier is sized and monitored for its own failure modes
• The route from custody to brokerage is visible and needs no application work

What it costs

• Two runbooks and two sets of SLOs
• People will still call both of them the vault
• The custodial tier remains a store worth attacking

Choose differently when. A greenfield estate whose upstreams can all mint accounts or accept federation should refuse the custodial tier and handle the few exceptions through the register.

Why it holds up over time. The distinction is a property of upstreams, not of tools. As upstreams add account creation or federation the custodial list shrinks, and nothing in the design changes.

Lesson. Two weaker promises kept honestly are worth more than one strong promise quietly broken for the secrets that matter most.

ADR-13 · Non-exportable secrets are transit keys; the platform returns operations, never material

Status: Accepted · Shown on views: 14, 32, 33

How does a workload use a signing or decryption key without ever holding it?

Context. Signing keys copied to build agents are among the most damaging leaks an organisation can have, because what they sign is trusted for years. The requirement asks that a secret declared non-exportable be exposed only as an operation.

Decision. A non-exportable secret is generated in, or imported into, the custody cluster's transit engine with export disabled. Workloads call sign, verify, encrypt or decrypt through the gateway under per-identity policy that separates signing from verifying. Break-glass access cannot read these keys either.

How it works on-premise. Transit key types chosen per consumer: Ed25519, ECDSA P-256, RSA-3072 and AES-256-GCM. Existing keys are imported with transit's wrapped import. Signing tools that support a Vault-compatible KMS provider, such as cosign, use it directly; the rest use a small adapter. Operations are rate-limited per key.

What it buys

• A compromised build agent can sign while it is compromised, and not afterwards
• Every signature and decryption is attributable
• Key rotation is invisible to callers

What it costs

• A network call per operation, and availability coupling for signing workloads
• Some libraries expect a key file and need an adapter
• Bulk decryption at high rates is not a good fit

Choose differently when. For high-volume data decryption inside a workload, issue per-object data keys wrapped by transit, and keep only the wrapping key non-exportable.

Why it holds up over time. An interface that says sign this is stable across algorithms. Moving to post-quantum signatures changes a key type and the verifiers, not the code path of every signer.

Lesson. The safest way to distribute a key is to distribute what it does.

Brokered issuance

How long credentials live, what ends them when nobody is watching, the credential that creates credentials, upstreams that cannot revoke, and protecting upstreams from the platform.

ADR-14 · Lease lifetimes: the platform sets bounds, roles set values, callers may only shorten

Status: Accepted · Shown on views: 19, 23, 28

How short is short, and who decides?

Context. A fifteen-minute lease contains a leak to fifteen minutes only if nobody renews it. While the legitimate holder is alive and renewing, a leaked copy lives as long as the total renewable lifetime. Shorter leases add renewal load, account churn and audit volume, and shorten how long workloads survive a control-plane outage. The requirement sets defaults of 30 minutes, 60 at most and 12 hours in total, and 15 minutes with no renewal for irrevocable roles.

Decision. The platform bounds are a 30-minute default, a 60-minute maximum and a 12-hour total, with 15 minutes and no renewal for irrevocable roles. Each role declares its own default, maximum and total within those bounds; owners of high-value roles set a shorter total. A caller may request less than the role's default and never more. The registry pull request shows the renewal rate, account churn and audit volume the chosen values imply.

How it works on-premise. OpenBao role default and maximum TTLs; the total lifetime is enforced by the session's 12-hour maximum and by the agent re-minting at the total. The compiler rejects out-of-bounds values and computes the load delta from the lifetime model on view 28.

What it buys

• Exposure is explained separately for a dead holder and a live one
• High-value roles can buy shorter exposure at a visible price
• Load is predictable from declared values

What it costs

• Variety in configuration that operators must understand
• Renewal load grows as owners shorten lifetimes
• The live-holder case still needs revocation and detection

Choose differently when. If leak detection proves precise, total lifetimes can lengthen for low-value roles. If an upstream accepts federation, the question disappears for that upstream.

Why it holds up over time. The model is arithmetic: renewals are outstanding leases divided by two-thirds of the lifetime. It holds for any engine, and the published table is regenerated from telemetry rather than argued from memory.

Lesson. A shorter lease buys nothing while its holder is alive. Know which exposure you are buying down before you pay for it.

ADR-15 · Leases hang off sessions, so a vanished workload's credentials expire with its session

Status: Accepted · Shown on views: 14, 19, 21

What revokes a pod's credentials when the pod disappears and nobody tells the platform?

Context. Pods are killed without warning, nodes lose power, and deletion events from Kubernetes are sometimes missed. The requirement asks for revocation when the identity a credential was issued to disappears. Any design that waits for a message saying so will sometimes wait forever.

Decision. Every lease is created under the session's OpenBao service token, and OpenBao revokes a token's leases when the token expires or is revoked. The agent renews a pod's session only while that pod's volume is mounted on its node. A pod that vanishes stops being renewed, and within 15 minutes its session and every lease under it are revoked. A clean unmount revokes immediately as a fast path.

How it works on-premise. Service tokens with a 15-minute TTL, renewable to 12 hours, one per pod instance; about 70 token renewals a second across the estate. Revocation of child leases runs in OpenBao's expiration manager and is confirmed at upstreams by the reconciler.

What it buys

• Node loss and pod loss are handled by the same mechanism
• Revoking one session contains everything issued through it
• Correctness does not depend on any event arriving

What it costs

• A window of up to 15 minutes after an unannounced disappearance
• Session revocation cascades, so a bug that revokes sessions breaks credentials broadly
• About 70 token writes a second across the estate

Choose differently when. If agents were ever replaced by application SDKs, those SDKs would have to renew sessions under the same contract, or this property would be lost silently.

Why it holds up over time. It is the dead-man's-switch pattern: the default outcome of silence is safety. The pattern survives any change of engine, orchestrator or agent that preserves the contract.

Lesson. Design so that failing to send a message produces the safe outcome. Deletion events get lost; silence does not.

ADR-16 · Upstream admin credentials are scoped to one upstream in one environment, rotated daily and never readable

Status: Accepted · proof required · Shown on views: 09, 11, 32

How is the credential that creates credentials kept from being the largest leak in the system?

Context. A broker must hold an account on each upstream that can create and delete accounts. That account is more valuable than everything it creates. The requirement calls these the crown jewels, asks for the tightest custody and shortest rotation, and forbids any one of them covering more than one environment.

Decision. Each broker connection uses its own admin account per upstream cluster per environment, with privileges to create, alter and drop platform-named accounts and to grant membership in pre-created privilege roles, and nothing else. On PostgreSQL 16 and later, an account with CREATEROLE manages only the roles it created. OpenBao rotates each admin password daily so no human knows it, and no read path returns connection configuration. The reconciler uses a separate account that can only list and drop platform-named accounts.

How it works on-premise. Privilege roles such as orders_rw are created by DBAs and granted to minted accounts; minted accounts carry VALID UNTIL and a connection limit. RabbitMQ admin users are limited to the platform's virtual hosts. A failed root rotation is recovered by a DBA pair re-seeding the account through break-glass.

What it buys

• Nobody, including platform operators, knows any admin password
• Compromise of one shard's memory is bounded to one environment's upstream family
• Orphan cleanup cannot create anything

What it costs

• DBA work to set up scoped admin and privilege roles on every cluster
• PostgreSQL 16 or later required for the tight CREATEROLE scoping
• A failed root rotation needs a two-person recovery

Choose differently when. For upstreams with no scoped administrative model, give them a broker shard of their own so their admin credential shares a memory image with nothing else, or keep them custodial.

Why it holds up over time. Least privilege for machine administrators, and a separate account for cleanup, are properties of the upstreams and the registry. They carry over unchanged to any engine.

Lesson. The admin credential of a credential factory should be able to make accounts, not use them.

ADR-17 · Irrevocable upstreams are brokered with 15-minute unrenewable leases and registered as risks

Status: Accepted · Shown on views: 11, 28, 30

What does revoked mean for an upstream that cannot revoke?

Context. Some internal token endpoints issue signed tokens with no revocation list, and some legacy services create keys they cannot delete early. The requirement's options are to refuse them, to broker with leases short enough that expiry is the containment, or to hold a credential custodially and rotate it after an incident.

Decision. Broker them, with a maximum of 15 minutes and no renewal, and mark the role irrevocable. Each such role has an exception register entry with an owner and a review date. Revoking an irrevocable lease stops re-issuance and is reported as expiring at a stated time, never as revoked.

How it works on-premise. Built plugins for internal token endpoints declare their revocation method; the compiler refuses a plugin with none unless the role carries the irrevocable flag. Agents re-mint at about ten minutes, and the upstream's churn budget is checked in the registry pull request.

What it buys

• An honest, known bound on every irrevocable credential
• Irrevocable upstreams are visible on the register rather than hidden in a success rate
• Owners have a reason to add revocation

What it costs

• Continuous re-minting load on the upstream
• A leaked irrevocable credential is usable for up to 15 minutes regardless of response
• Brief interruptions if minting is unavailable for longer than a lease

Choose differently when. If an upstream cannot tolerate the mint rate that 15-minute leases imply, custody with rotation is the lesser evil, registered with its slower containment stated.

Why it holds up over time. Entries close as upstreams add revocation or federation, and the class shrinks without any design change.

Lesson. When you cannot take something back, give less of it for less time, and write down that you did.

ADR-18 · Per-role lease caps and per-upstream mint budgets shed issuance before an upstream is exhausted

Status: Accepted · Shown on views: 22, 28, 34

What stops the platform from taking down an upstream it exists to serve?

Context. A full node-pool replacement asks for hundreds of thousands of new accounts in minutes. PostgreSQL creates roles at a few hundred a second per primary, and every minted credential that connects consumes a connection slot. Seven hundred roles share fourteen kinds of upstream. The requirement asks for caps per role, shedding before exhaustion, and partitioning so one high-churn role cannot starve others on the same upstream.

Decision. Each role declares a lease cap, by default three times its expected instance count. Each upstream declares a mint budget per second and a connection budget. The gateway enforces both: per-role concurrency from the broker's outstanding-lease metric, refreshed every ten seconds and applied at 90% of the cap, and a token bucket per upstream. Requests over budget receive a retryable 429 naming the upstream. The renewal lane is never subject to mint budgets.

How it works on-premise. Mint budgets start at 200 role creations a second per PostgreSQL primary and are tuned from the proof phase. The OpenBao database plugin's connection pool is capped per upstream. Minted PostgreSQL roles carry a connection limit set by the role declaration.

What it buys

• A restart storm slows new starts instead of breaking running databases
• One noisy role cannot exhaust an upstream shared with others
• Budgets are declared data that the upstream's owner can review

What it costs

• Budgets need tuning per upstream and drift as fleets grow
• Metric-based caps are approximate within a ten-second window
• New pods of a capped role wait during storms

Choose differently when. Upstreams that federate workload identity have no accounts to create, and their budgets can be removed.

Why it holds up over time. Admission control that protects a shared dependency is independent of the engine. The budgets are data in the registry and move with the upstream, not with the platform.

Lesson. A platform that can issue faster than its dependencies can absorb must be the first to say no.

Delivery

What runs next to the workload, how a credential reaches a process, what happens when the control plane is gone, and time.

ADR-19 · One credential agent per node, built, using SPIRE's delegated identity and holding everything in memory

Status: Accepted · proof required · Shown on views: 08, 17, 22

What runs next to the workload to obtain, renew and deliver its credentials, and how many copies of it are there?

Context. The requirement asks for a local agent so that applications never handle attestation, renewal or the platform's address, never write to disk, never extend a lifetime, notify on change, and continue from memory during an outage. The existing choices are OpenBao Agent or Proxy as a sidecar in every pod, the Secrets Store CSI driver with a provider, or client libraries in each language.

Decision. Build one Go agent per node, deployed as a DaemonSet. It is a CSI node plugin that mounts a per-pod socket directory and a per-pod tmpfs volume; a registered delegate of the SPIRE agent, obtaining each local pod's SVIDs through the Delegated Identity API; the session and lease manager; an SDS server for proxy mode; and the enforcer of monotonic ageing, the skew guard, jittered attestation and change notification.

How it works on-premise. Pages locked with mlock and the process marked non-dumpable. Change notification through a watch stream on the socket and atomic rename for files. Rolled out through Argo CD to 1%, 10% and then all nodes, with the gateway supporting the current and previous agent protocol.

What it buys

• The invariants are enforced by one program on every node
• Per-node blast radius is enumerable and matches the node's own
• Applications code against a local socket and nothing else

What it costs

• A security-critical daemon on every node, owned by the platform team
• Correctness depends on the SPIRE agent's attestation of local pods
• A defect affects every pod on a node, which demands staged rollout

Choose differently when. An estate of a few hundred pods should use OpenBao Agent sidecars; the cost of building and owning a node agent is not repaid at that size.

Why it holds up over time. The socket protocol is the contract applications depend on, so it is the one interface to keep backwards compatible. Gateways, engines and SPIRE can all change behind it.

Lesson. Put the hard invariants in one small program you own, next to the workload, rather than in forty libraries you do not.

ADR-20 · Three delivery modes, the socket first; environment variables are refused

Status: Accepted · Shown on views: 04, 07, 08

Where does the credential land: an endpoint the application calls, a file it reads, or a proxy that uses it on the application's behalf?

Context. An endpoint is rotation-native but requires application change. A file works unmodified but is readable by anything in the container and makes rotation a file-watch problem. A proxy gives the strongest containment but only for protocols it can speak. Environment variables are the most common choice in practice and the worst: inherited by every child, printed in crash dumps and frozen until restart.

Decision. Offer three modes permanently. The local socket is the default. A tmpfs file serves software that cannot change, and such workloads are registered as unmodifiable consumers and excluded from routine short-lease rotation until they change. An Envoy credential injector serves HTTP APIs whose key the application should never see. Database wire protocols are not proxied. Environment variables are refused: the agent has no such mode, and admission policy rejects pods that reference platform paths through the environment.

How it works on-premise. The socket speaks HTTP over a unix domain socket with a watch endpoint. Files are written by atomic rename, 0400, owned by the pod's UID, removed on unmount. Envoy receives credentials over SDS from the agent. Admission rules run in the estate's existing admission controller.

What it buys

• The safest mode is the easiest one
• Legacy workloads are supported and named, not ignored
• An application in proxy mode cannot leak a key it never had

What it costs

• Three delivery paths to test and support indefinitely
• Framework integrations must be written for the socket
• Teams used to environment variables need migration help

Choose differently when. An estate made almost entirely of unmodifiable vendor software would make file delivery the default and invest in consumer-side reload hooks.

Why it holds up over time. Each mode uses a primitive older than the platform: unix sockets, tmpfs and SDS. They will outlast any agent implementation.

Lesson. Offer the unsafe but necessary path explicitly, name who uses it, and remove the one that is merely convenient.

ADR-21 · Fail static until expiry; no grace extension exists anywhere

Status: Accepted · Shown on views: 19, 23, 34

What should agents do when the control plane is unreachable?

Context. The options are to serve cached credentials until they expire, to extend them for a bounded grace period, or to fail closed at once. Serving to expiry turns a long outage into rolling estate-wide failure. A grace period prevents that, but at the moment of decision an outage and a compromise look the same, and every containment claim becomes conditional on nobody having used the grace. Failing closed at once is defensible only if the platform is more available than everything that depends on it.

Decision. Agents serve valid cached credentials until their own expiry and not a moment longer. No agent setting, gateway flag or operator command extends a lifetime. Custodial copies are evicted 60 minutes after they were read. Availability is bought through the control plane: 99.99% issuance per site, five voters in three rooms, a 10-minute lease-store RTO and quarterly outage drills.

How it works on-premise. The agent's cache has no code path that alters an expiry, and the invariant suite runs every failure mode against it and fails the build if any credential is served past its deadline.

What it buys

• Containment claims hold during outages as well as in steady state
• No one can be pressured into extending lifetimes during an incident
• The outage profile is known in advance and drilled

What it costs

• After 60 minutes of site outage, credential-dependent workloads at that site fail
• The platform's availability target becomes the estate's ceiling
• The degradation must be explained to every service owner

Choose differently when. A workload whose own availability target exceeds the platform's should not depend on brokered credentials. It should use federation where the upstream supports it, or a registered custodial credential with a long maximum age.

Why it holds up over time. The absence of a switch is the most durable control there is. No future incident commander can flip what was never built, whatever the pressure.

Lesson. Never build a switch you would be tempted to flip under pressure if flipping it silently falsifies your guarantees.

ADR-22 · Agents age credentials on a monotonic clock and refuse to serve beyond ±30 s of skew

Status: Accepted · Shown on views: 08, 19, 34

How does a node with a wrong clock avoid serving an expired credential or refusing a valid one?

Context. Wall clocks jump: NTP corrections, virtual machine pauses, a mis-set BIOS after hardware replacement. An agent that compares an absolute expiry with a wrong clock serves credentials that have expired or discards ones that have not. The requirement sets a ±30 s tolerance and asks that an agent beyond it refuse to serve.

Decision. Every credential is issued with a TTL and an absolute expiry. The agent computes its deadline as the monotonic time of receipt plus the TTL minus the round-trip time, and uses that to decide whether to serve. The wall clock is used only for file-mode consumers that read an absolute expiry and to compare with gateway time and chrony status. Beyond ±30 s of skew the agent stops serving, reports it, and the node is cordoned.

How it works on-premise. chrony on every node against the site's stratum-1 sources, skew exported as a metric, and a node controller that cordons on the agent's report.

What it buys

• Expiry is enforced correctly on nodes with bad clocks
• Clock faults surface as named node problems
• The agent and the upstream agree on when a credential ends

What it costs

• A node with a bad clock loses credential service until fixed
• Another metric and controller to operate
• Round-trip estimation makes deadlines slightly conservative

Choose differently when. None likely. The alternatives are strictly worse, and the only variable is the tolerance, which can be tightened if clock quality allows.

Why it holds up over time. Monotonic clocks exist on every operating system the agent will ever run on, and the rule does not depend on any product.

Lesson. Measure durations with a stopwatch, and ask a clock only what time it is.

Rotation and revocation

Who owns the last mile of rotation, what contained means, how two systems of record are kept honest, and how a restore avoids reviving the past.

ADR-23 · Rotation keeps two working credentials and retires on observed use, with a seven-day ceiling

Status: Accepted · Shown on views: 06, 20, 34

Who owns the last mile of rotation, and how long may a straggler hold the old credential?

Context. Pushing a new credential into consumers requires reaching into every one of them. Waiting for observed use to stop is safe but unbounded, because one straggler keeps the old credential alive indefinitely. Retiring on schedule gives the window meaning and guarantees breakage the day a consumer is missed. The requirement sets a 24-hour overlap and retirement after 24 hours with no observed use.

Decision. Automated custodial rotation keeps two working credentials: alternating accounts for databases, two keys for APIs that allow it. The new version is staged at both sites, verified with a real call, and promoted. The old version is retired after 24 hours with no observed use. If it is still used on day seven, it is retired anyway and the named straggler breaks, after notices at 24 hours and five days. A secret may declare retire-on-schedule instead. Emergency rotation skips the overlap entirely.

How it works on-premise. The rotation controller in Go, with one rotator per upstream kind. Observed use combines agents' reports of the version each workload holds with the upstream's own evidence: login logs per account, or a partner's usage report per key. A rotation that fails verification aborts and leaves the old version active.

What it buys

• Rotations never leave zero working credentials
• Stragglers are found by name, including consumers outside the agent
• The ceiling gives the overlap a meaning without guaranteeing breakage

What it costs

• Upstreams must support two concurrent credentials
• A straggler will break on day seven, deliberately
• Observed use depends on upstream logs being available

Choose differently when. Where a partner allows only one active key, rotation is manual with a named owner, and every rotation is effectively an emergency one.

Why it holds up over time. Dual-credential rotation is a property of how upstreams authenticate, not of the platform. The ceiling is a number in the registry that can move as consumers improve.

Lesson. A window that never closes is not a window. Give stragglers a date, and make sure they hear it early.

ADR-24 · Revocation by scope resolves to entries, sessions and leases, and reports contained only when the upstream confirms

Status: Accepted · Shown on views: 05, 21, 34

How does a responder revoke exactly the right set of credentials, and when may the platform say the job is done?

Context. Incidents rarely involve a single credential. The requirement asks for revocation by lease, identity, role, node, namespace and everything, propagated to upstreams within 30 s at p99 and within five minutes at worst, and for failure to be reported as a live, uncontained credential rather than as success.

Decision. Each scope resolves through the inventory. An identity scope disables its SPIRE entry and revokes all its sessions, taking their leases with them. A node scope evicts the node's SPIRE agent and revokes sessions issued through it. A role scope revokes the role's lease prefix and disables the role. A namespace scope is the union of its identities. Everything closes the partition at the gateway and revokes every prefix. New issuance is stopped before live credentials are revoked. A lease is contained only when the reconciler reads the upstream and finds the account gone; after five minutes it is reported uncontained and pages until closed.

How it works on-premise. Revocation is synchronous in OpenBao. PostgreSQL revocation statements set NOLOGIN, terminate the role's backends, then drop the role. The inventory previews the workloads in scope before execution.

What it buys

• Responders revoke by the question they are asking, not by lease ID
• The report says what was observed, not what was requested
• Unaffected workloads re-mint within seconds

What it costs

• Revocation depends on inventory freshness for scope resolution
• An upstream that is down leaves leases reported as uncontained until it returns
• Wide scopes create re-mint bursts that the admission lanes must absorb

Choose differently when. Upstreams that accept workload identity directly need no account revocation at all; disabling the identity is the whole revocation.

Why it holds up over time. Stop new, end live, verify at the source is the pattern for every revocation system. The scopes are data in the inventory and do not depend on the engine.

Lesson. Report what you observed, not what you asked for.

ADR-25 · Orphaned accounts are reconciled hourly against upstream account lists

Status: Accepted · Shown on views: 13, 19, 24

How does the platform find a credential that is live at an upstream and unknown to it, or a lease that refers to an account that no longer exists?

Context. The lease store is the platform's record of what is outstanding, but a crashed plugin call, a restored snapshot or a DBA's manual change can make it wrong. A lost lease record is a live account nobody will revoke, which the requirement calls the platform's worst data-loss outcome. It asks for reconciliation at least hourly and an orphan lifetime of at most 90 minutes.

Decision. A reconciler per broker shard lists platform-named accounts at each upstream every 60 minutes, staggered, and compares them with outstanding leases. An account with no lease is revoked through the reconciler's own list-and-drop account. A lease with no account is closed. Accounts without the platform prefix are reported and never touched.

How it works on-premise. Minted usernames carry a fixed prefix and the lease hash, which makes the comparison a set difference. Results go to the audit stream and a containment dashboard; a sudden rise in orphans is itself alerted on.

What it buys

• Orphans have a hard maximum lifetime
• Restores and plugin crashes are corrected without human intervention
• Manual account changes by others become visible

What it costs

• Hourly list queries against every upstream
• Up to 90 minutes of orphan life
• Reconciler accounts to scope, rotate and audit

Choose differently when. If an upstream can tag accounts with the lease that created them and expire them itself, reconciliation for that upstream becomes a consistency check rather than a cleanup.

Why it holds up over time. Reconciling two systems of record is the oldest control in operations, and it works with any engine and any upstream that can list its accounts.

Lesson. Two systems of record that disagree give you a finding. One that nobody checks gives you a belief.

ADR-26 · A restore advances a generation held outside the restored store, and revalidates before serving

Status: Accepted · proof required · Shown on views: 12, 13, 24

How is a store restored from backup without reviving a revoked credential, an attacker's session or a rotated-out secret?

Context. A snapshot is a copy of the platform's past decisions. Restoring one brings back sessions that were revoked, leases whose accounts were dropped and custodial versions that were rotated out. The requirement asks that restored leases be revalidated against upstreams and that a restore advance a generation so pre-restore sessions are not honoured.

Decision. Each partition has a generation stored in the KMS cluster, which is never restored together with the engines it protects. Restoring a broker or custody cluster requires an SRE and a key custodian to advance the generation. Gateways refuse every session from an earlier generation and keep the partition closed until the reconciler has closed leases with no account, revoked accounts with no lease, and compared custodial versions with the rotation ledger held in the audit store. Secrets whose restored version is older than the ledger says are quarantined until rotated forward.

How it works on-premise. Raft snapshots every 15 minutes to the site's backup store, encrypted by the barrier. The restore runbook is a script with two approvals. It is drilled quarterly in staging against a snapshot taken 40 minutes before a burst of revocations.

What it buys

• No session from before an incident survives a recovery
• Revoked accounts and retired versions cannot return to service
• Restores are safe to rehearse

What it costs

• Every workload in the partition re-attests after a restore
• Revalidation lengthens RTO and must be proven inside ten minutes
• Two people are needed to do something that is technically quick

Choose differently when. A platform whose broker state is small enough to rebuild empty within the RTO, and whose upstreams tolerate mass re-creation, can prefer rebuilding to restoring.

Why it holds up over time. Generation counters and revalidation after restore are how consensus systems and certificate authorities have handled the same problem for decades. The generation's location outside the restored store is the part that must never change.

Lesson. A backup of a security system is a copy of its past decisions. Restore it only with a way to overrule them.

Governance, audit and people

How policy is made, how blast radius is known, how reads are recorded and kept, how a human reads a secret, and how leaks are noticed.

ADR-27 · Policy is compiled from a reviewed registry; nobody writes policy

Status: Accepted · Shown on views: 04, 14, 16

How are grants made, reviewed and reverted so that the policy in force is exactly what people approved?

Context. The requirement forbids inheritance, wildcards at the top of a path and default grants; asks for a second approval for production custodial grants; asks for policy to be versioned, validated against known-good requests and revertible in one operation; and requires acknowledgement before any change that would deny a currently succeeding production identity. Hand-written engine policy, however carefully reviewed, is reviewed in the wrong language.

Decision. Teams declare identities, secrets, roles and grants in the registry. The compiler rejects wildcards of any kind, grants without an owner and environment mismatches; emits OpenBao policies, JWT roles and SPIRE entries; replays seven days of real requests from the audit store against the result; and computes each identity's reachable set. Production custodial grants and changes that deny current traffic need two approvers, neither of them the author. A single applier identity per environment writes the compiled bundle. Revert is a Git revert through the same pipeline with approval waived.

How it works on-premise. Forgejo with protected branches and code owners, Forgejo Actions for compilation and replay, a signed bundle as the only input to the applier, and no human policy-write role in production.

What it buys

• Every grant has an owner, an approver and a commit
• A bad push is caught by replay before it applies, and reverted in minutes if not
• Nobody in production can change policy by hand

What it costs

• Grant changes take a pipeline run rather than a click
• The compiler and replay are platform code to maintain
• The applier identity is powerful and must be guarded

Choose differently when. An organisation already standardised on a request-time policy engine across its estate might keep that engine and compile the registry into it, keeping the environment boundary in the handshake.

Why it holds up over time. The registry outlives engines. Changing the engine or its policy language changes the compiler, while ten thousand approvals remain valid as written.

Lesson. Let people declare intent in the smallest language that can be reviewed, and let a machine write the language that is enforced.

ADR-28 · Blast radius is computed from compiled policy, per identity, per node and per shard

Status: Accepted · Shown on views: 04, 21, 32

How does the platform know, for anything that could be compromised, exactly what that compromise would yield?

Context. The requirement makes the enumerated reachable set the definition of blast radius, per identity and per node, and asks for review of any identity above a threshold, because a workload that needs very many secrets is usually several workloads.

Decision. On every apply, the inventory computes each identity's reachable set from the compiled grants, each node's set from the pods scheduled on it, and each broker shard's set from its roles and admin credentials. Sets are published, shown in pull requests and used to preview revocation scopes. An identity reaching more than 40 secrets and roles is flagged for review. A separate report lists grants unused for 90 days, from the audit store, so reachable sets shrink over time.

How it works on-premise. The inventory service in Go on PostgreSQL, reading compiled policy and the estate's pod schedule every five minutes.

What it buys

• Every blast-radius claim in this record can be checked by a query
• Over-granted identities surface at review time, not during an incident
• Responders scope revocation with real numbers

What it costs

• Another service on the governance plane
• The node view is as fresh as its five-minute schedule read
• Threshold reviews create work for teams with legitimately broad needs

Choose differently when. None likely. The alternative is not knowing, and that is not a position the requirement allows.

Why it holds up over time. It is a join over identities, grants and placement. The computation survives any engine or orchestrator that can say who is granted what and what runs where.

Lesson. If you cannot list what a compromise yields, you do not know your security posture; you have an opinion about it.

ADR-29 · Audit posture is declared per event class: synchronous quorum for custody and humans, local-durable and shipped for renewals

Status: Accepted · Shown on views: 12, 15, 18

When the audit sink is unavailable, which requests must be refused and which may continue?

Context. The requirement asks that every read be recorded before the material is returned; that custodial reads and human access fail closed when the audit sink is unavailable; and that high-volume brokered renewals fail open with a recorded gap. OpenBao's audit devices are all or nothing: if every device fails, every request fails.

Decision. For custodial reads, human access and policy changes, the gateway writes the record to Kafka with acks=all and returns material only after the acknowledgement; without it the request fails and the material is zeroed. For mints and renewals, OpenBao writes to a local audit log on NVMe before replying, which it will not skip, and Fluent Bit ships the log to Kafka; if shipping stalls or its disk buffer overflows, a gap marker with the missing range is written as soon as the stream is reachable. The KMS keeps its own audit devices for every unwrap.

How it works on-premise. Kafka in KRaft mode, three brokers per site, replication factor three and minimum in-sync replicas two. Local audit logs on dedicated NVMe with 72 hours of capacity. Fluent Bit with a filesystem buffer.

What it buys

• No custodial or human read completes without a durable, off-host record
• An audit outage does not stop running workloads
• Gaps are records in themselves, not silent losses

What it costs

• Kafka is on the custodial read path
• Brokered events can reach the immutable store up to 30 seconds late
• Two audit paths to reason about and test

Choose differently when. If custodial read volume became very high and Kafka's latency a problem, the synchronous write could go to a local quorum log on the gateway hosts, shipped onward, with the same fail-closed rule.

Why it holds up over time. Declaring a failure posture per class of event is a requirement-level decision. It remains true whatever carries the events.

Lesson. Decide in advance which reads you would rather refuse than leave unrecorded.

ADR-30 · Audit lands in ClickHouse for 400 days and in Ceph Object Lock for seven years, owned by the records team

Status: Accepted · Shown on views: 13, 15, 31

Where is the audit record kept so that it can be queried quickly, retained for years and altered by no platform operator?

Context. The requirement asks for 400 days queryable, seven years archived, human access records immutable for seven years, a single query answering who read a secret in a window, and a store the platform's operators cannot alter. At about 13 million brokered events a day per site, the renewal stream's retention is also the largest single cost.

Decision. Kafka feeds ClickHouse, ordered by secret path and time with a projection by actor, with a 400-day TTL. A chain sealer writes hourly hash-chained Parquet segments per partition to a Ceph RGW bucket with Object Lock in compliance mode for seven years. The records team administers ClickHouse and the Ceph cluster; the platform team holds producer rights only. Audit retention is reported as its own cost line.

How it works on-premise. ClickHouse ReplicatedMergeTree on three nodes per site with ClickHouse Keeper. A dedicated Ceph cluster for records at site B, replicated to site A. Chain anchors are also printed into the monthly security report so an edit would have to rewrite a document outside the system.

What it buys

• The first question of every incident answers in seconds
• Evidence survives the platform and its operators
• Tampering with a segment breaks a chain whose anchors live elsewhere

What it costs

• Two storage systems owned by another team
• Seven years of compliance-mode retention cannot be shortened, even by mistake
• Renewal events dominate storage and must be watched

Choose differently when. An organisation with an existing audit log service that meets these retention and immutability rules should send events there and not build a second one.

Why it holds up over time. Hash-chained Parquet in S3-compatible object storage is readable by any analytics tool of the next decade. The record does not depend on ClickHouse, Kafka or this platform existing.

Lesson. Evidence that its subject can edit is not evidence. Give it to someone else to keep.

ADR-31 · No standing human read; break-glass is two-person, reasoned, response-wrapped and single use

Status: Accepted · Shown on views: 31, 32, 33

How does a human ever read a secret, and how is the platform's own staff treated as a threat?

Context. The requirement treats operators as an in-scope threat: no standing read access to material, break-glass with second authorisation, a stated reason, a time bound and an incident-grade record, and no operator credential able both to read material and to delete the record of reading it.

Decision. No human role has a policy on any data path. Platform SREs hold system operations only; security responders hold revocation and metadata. A human read goes through the access broker: sign-in with WebAuthn, a request naming the secret, the reason and the incident, a synchronous record, approval by a different person in the approver group, then a response-wrapped read valid once for 15 minutes. The unwrap pages security. Every break-glass read schedules an emergency rotation of that secret.

How it works on-premise. Keycloak federated with the corporate identity provider for operators, with hardware-key WebAuthn required. The access broker in Go on the governance cluster, with its own OpenBao identity allowed only to create wrapped reads. Non-exportable transit keys cannot be read by this path at all.

What it buys

• A compromised operator account causes an outage at worst, never a read
• An intercepted wrapping token announces itself
• Every human read ends in a rotation

What it costs

• Human access takes minutes and two people
• Emergency rotations follow every break-glass read
• The access broker is on the path of every incident that needs material

Choose differently when. None likely for production. In development partitions a single approver is acceptable and recorded.

Why it holds up over time. Two-person control, a stated reason and a single use are organisational rules that have outlived every access tool that has implemented them.

Lesson. Make the human read expensive, rare and loud, and assume that whatever was read is gone.

ADR-32 · Leak detection joins upstream logs to leases; decoys are real accounts with no privileges

Status: Accepted · Phase 3 · Shown on views: 05, 15, 29

How does the platform notice a leaked credential that its legitimate holder is still renewing?

Context. Expiry contains a credential whose holder has died. A credential whose holder is alive is renewed, and a leaked copy stays valid for up to 12 hours. The requirement asks for detection of use inconsistent with the workload a credential was issued to, and for decoy credentials whose use is a confirmed incident.

Decision. Upstream authentication logs (PostgreSQL connection logs, RabbitMQ authentication events, partners' key-usage reports) are shipped to ClickHouse and joined on username to lease, session, pod and node. An alert fires when a credential is used from an address that is not its holder's pod, after its lease closed, or from outside the environment's networks. Decoy identities mint real accounts that can log in and have no privileges, and their credentials are placed where a thief would look. Any login with a decoy is a confirmed incident.

How it works on-premise. Log shippers on upstream hosts with the owners' agreement, a daily join of pod addresses from the estate's IP address management, and decoys refreshed monthly so they look current.

What it buys

• Live-holder leaks become detectable
• Decoys produce findings with no false positives
• The same join answers where was this credential used during an investigation

What it costs

• Integration with every upstream's logging
• Address attribution breaks down behind NAT and connection poolers
• Decoys must be maintained so they stay believable

Choose differently when. Upstreams that accept workload identity directly can check the presenter's identity themselves, which makes this join unnecessary for them.

Why it holds up over time. Matching use to issuance is independent of any product, and the join keys (usernames that encode leases, pod addresses) are under the platform's control.

Lesson. Expiry contains the leaks you never learn about. Detection is for the ones still being renewed.

Operations and cost

How the platform changes without becoming the outage, and how the cost of a security choice is put in front of the person making it.

ADR-33 · Releases go partition by partition and site by site, gated by an invariant suite

Status: Accepted · Shown on views: 26, 29, 34

How does a platform on every workload's start-up path change without becoming the outage?

Context. A defect in the gateway, the agent or an engine upgrade can stop every new pod in an environment. The invariants that matter most (no lifetime extension, no environment variables, no material in logs) are exactly the ones a well-intentioned change breaks without any functional test noticing.

Decision. Every release passes the invariant suite, a material scrubber over logs, traces and error messages, and a storm replay. It then soaks for 48 hours in staging at both sites, goes to one production partition at site A one node at a time, and reaches site B only after 24 hours clean. Agents roll out to 1%, 10% and then all nodes. No production change happens during a SEV1 or SEV2, or during a scheduled estate-wide redeploy.

How it works on-premise. Reproducible Go builds in Forgejo Actions, cosign signatures and SBOMs, Ansible for enclave hosts with signed artefacts staged on-host before the change window, and Argo CD for the agent DaemonSet. The previous binaries stay staged for a one-command rollback.

What it buys

• A defect is found in one partition, not everywhere
• Invariants are guarded by tests rather than by memory
• Rollback needs no network fetch

What it costs

• A release takes several days to reach every partition
• Staging must be realistic enough for a soak to mean something
• Change freezes during incidents can delay fixes

Choose differently when. Security fixes for actively exploited vulnerabilities follow an expedited path with the same order and shorter soaks, approved by the security lead.

Why it holds up over time. Staged rollout and property tests are practices, not products. They carry across any change of language, engine or deployment tool.

Lesson. The properties that make a platform safe are the ones ordinary tests do not check. Write tests for them by name.

ADR-34 · Lifetime is published as load and cost, shown rather than charged, and cache hit rate is a headline metric

Status: Accepted · Shown on views: 22, 27, 28

How are the costs of security choices made visible without punishing teams for choosing safely?

Context. Lease lifetime is the platform's main load parameter. Shorter leases mean more renewals, more account churn upstream and more audit rows, and the renewal stream's retention is often the largest cost in the system. The requirement asks for attribution per team, environment and secret, the cost implied by each lifetime, upstream cost as well as platform cost, audit retention as its own line, and cache hit rate reported with cached reads free.

Decision. The lifetime model publishes renewals, mints, account churn, audit rows and storage for every role's chosen values, and the registry pull request shows the change. Monthly reports attribute requests, mints and audit volume per team, environment and secret, including upstream accounts and connections consumed. Costs are shown, not charged. Cache hit rate sits on the platform's headline dashboard, and reads served from cache cost nothing.

How it works on-premise. Attribution is computed in ClickHouse from the audit stream joined to the registry's ownership data. Unit costs come from the enclave's hardware and power share, and upstream costs are supplied by the upstreams' owners.

What it buys

• Security and cost trades are made with numbers in front of the person making them
• Retention of renewal events is visible as the cost it is
• Clients bypassing the agent show up as falling cache hit rates

What it costs

• Attribution pipelines to maintain
• Showback without chargeback relies on steering-group attention
• Unit costs for on-premises hardware are estimates and invite argument

Choose differently when. An organisation with mature internal chargeback may charge per team, provided it charges for audit retention and upstream consumption rather than for request count.

Why it holds up over time. The model is arithmetic over declared lifetimes and observed volumes. It stays correct as prices, hardware and engines change, and is regenerated from telemetry rather than rewritten.

Lesson. Show people the cost of a security choice when they make it, and never charge them for choosing safely.

Every package used, in one table

Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.