Architecture Views

What does a workload present to prove what it is, and where does that chain of proof finally end?

Context

Every secrets platform has to answer secret zero: the credential a workload uses to obtain its credentials. If the answer is a token injected by the pipeline or baked into an image, the platform has concentrated the estate's secrets behind one more valuable secret. The requirement asks for an attestation produced by the infrastructure, verified against the issuer's published keys, bound to cluster, namespace, service account and image digest, refused for unregistered coordinates, and a register for every place where no attestation is possible. On premises there is no cloud identity federation to lean on. What the organisation does own is a Kubernetes control plane that already signs service-account tokens and a kubelet that already knows what each container is.

Decision

Use SPIFFE as the identity contract and SPIRE as its implementation. Kubernetes nodes attest with the SPIRE agent's projected service-account token, verified by TokenReview against the cluster's API server. Pods are attested by the SPIRE agent through the kubelet on namespace, service account and image digest. The attestation presented to the platform is a five-minute JWT-SVID, sent over mTLS authenticated with the same pod's X.509-SVID. Bare-metal and VM hosts move to TPM DevID node attestation in Phase 3; until then a VM joins with a single-use join token that is a registered secret-zero exception.

How it works on-premise

Three SPIRE servers per environment per site on a Patroni-managed PostgreSQL datastore, each deployment holding a site intermediate signed yearly by the environment's offline root in an HSM. SPIRE agents run as a DaemonSet in every estate cluster. The API server service-account signing keys live on control-plane hosts with TPM-bound disk encryption and are registered as trust anchors, with their own custody and rotation procedure.

Options weighed

• ChosenSPIFFE identities issued by SPIRE: Attests nodes and workloads from what the infrastructure observes, supports several attestation methods behind one identity format, and is a CNCF graduated project under Apache 2.0.
• RejectedKubernetes service-account tokens sent straight to OpenBao: Simple and built in. No image digest, no node attestation, only one substrate, and the API server is called on every login.
• RejectedA bootstrap token injected by the delivery pipeline: Familiar from AppRole designs. It moves secret zero into the pipeline, which then becomes the one system whose compromise yields everything.
• DeferredTPM-rooted attestation for every host from day one: The strongest root. It excludes shared CI hosts and VMs without a virtual TPM, so it arrives for bare metal in Phase 3.
• Right elsewhereCloud workload identity federation: The requirement's own answer on Google Cloud, and right wherever a managed substrate is the trust root. Excluded by the on-premises constraint.

Consequences

What it buys

• No image, repository, pipeline or configuration file holds anything a workload uses to authenticate
• Identity describes what is running, including its digest, rather than what someone declared
• One identity format that the service mesh, databases and future federation can also consume

What it costs

• The API server's signing key becomes a trust anchor that must be guarded like a CA key
• SPIRE becomes a Tier 0 dependency with its own datastore, upgrades and drills
• VMs carry registered exceptions until TPM attestation is in place

Choose differently when

If most workloads ran on VMs or bare metal rather than Kubernetes, TPM node attestation should be the MVP path and service-account tokens the addition. If the estate lived on a single managed cloud, the provider's workload identity federation is the better root and SPIRE is optional.

Why it holds up over time

A SPIFFE ID is a URI and an SVID is an X.509 certificate or a JWT; both are open standards consumed by Istio, Cilium, Envoy and others. A later change of implementation, or of attestation method from tokens to TPMs, changes how an SVID is obtained and not what it says, so grants, audit history and dashboards remain valid.

How does a workload become known to the platform, and what stops an unknown one being trusted because it looks plausible?

Context

SPIRE's Kubernetes controller can create an identity for any pod that matches a template, which is identity on first sight: anyone able to create a pod in a matching namespace gets one. The requirement forbids that. It also asks that identity be bound to the image digest where the substrate supports it, but digests change on every build, so a registry of digests maintained by hand would be wrong within a day.

Decision

Identities are declared in the registry with environment, cluster, namespace, service account and owner. The compiler emits one exact SPIRE entry per identity and one OpenBao JWT role with an exact bound subject. The image-digest selector is written into the entry by the delivery pipeline at deploy time, from a signed image that admission control has accepted. An image not deployed by the pipeline has no matching entry, and its pod receives no identity.

How it works on-premise

The registry is a Forgejo repository of YAML. The compiler runs in CI and renders ClusterStaticEntry objects for spire-controller-manager, applied by Argo CD. The release job patches the digest selector for the workload it is deploying, using an identity allowed to change only that selector.

Options weighed

• ChosenRegistry-compiled static entries with pipeline-supplied digests: Identity requires a reviewed declaration, and the digest is always the one that was actually deployed.
• RejectedTemplated entries matching every pod in a namespace: No registry to maintain. Deploy rights become identity-creation rights.
• RejectedNamespace and service account without a digest: Survives rebuilds with no pipeline integration. Anything able to run a pod under that service account inherits the identity, including a debugging image.
• DeferredSignature-based selectors checked at attestation: Better long term, because a signed rebuild keeps its identity without a pipeline write. Revisited once SPIRE's signature selectors are stable on the chosen release.

Consequences

What it buys

• No identity exists that a person did not approve
• A hotfix pushed around the pipeline surfaces as a clear denial, not as a silent new identity
• Every identity has an owner who is paged when it is over-granted or orphaned

What it costs

• Every deploy writes to SPIRE, so the pipeline is on the path to a new version's first credential
• Images pushed by hand in an incident fail until run through the pipeline
• A slow registry review becomes a slow launch

Choose differently when

A small estate with a handful of teams and a strict admission controller can accept namespace and service-account templates, trading the registry's overhead for a boundary it can still reason about.

Why it holds up over time

The registry's schema names concepts (identity, secret, role, grant, owner), not SPIRE or OpenBao objects. If either product is replaced, the compiler changes and the approvals do not.

What does a workload hold between attestation and each request, and what stops it being useful to anyone else?

Context

OpenBao tokens are bearer tokens. Address binding exists but is not key binding, and pods change addresses. The requirement asks for session tokens bound to the client's key, 15 minutes long and renewable to 12 hours, and for a store restore to invalidate every session issued before it. X.509-SVIDs rotate their keys every 30 minutes, so any binding must follow rotation without a new attestation.

Decision

The gateway wraps the engine's token into a session token: an authenticated-encryption envelope holding the OpenBao token, the SPIFFE ID, a SHA-256 thumbprint of the presenting SVID's public key, the partition generation and an expiry. On every request the gateway checks that the TLS client key matches the thumbprint and that the generation is current. The agent renews the session about every ten minutes over its current SVID, which rebinds the thumbprint to the new key.

How it works on-premise

The session key is an AES-256-GCM data key issued by the KMS, rotated daily and held only in gateway memory, with the previous key accepted for one rotation. Gateways read the partition generation from the KMS every ten seconds.

Options weighed

• ChosenGateway-wrapped, key-bound session: A copied token is useless without the private key that never leaves the agent, and the gateway stays stateless.
• RejectedReturn the OpenBao token to the agent: No extra code. Any workload in the environment could replay a token that leaked from another.
• RejectedLog in to OpenBao on every request: No session to protect. Every renewal becomes a Raft write and a key-set validation, multiplying write load several times over.
• Right elsewhereDPoP proof on every request: The right tool for clients crossing untrusted networks without mutual TLS. Here mTLS already proves possession of the key on every connection at no per-request cost.

Consequences

What it buys

• A session token found in a log, a dump or a packet capture cannot be used
• Restoring a store invalidates every earlier session in one operation
• The gateway holds no per-session state and scales by adding instances

What it costs

• A token format owned by the platform, which needs fuzzing, versioning and review
• The session key is a secret in gateway memory, derived from the KMS and rotated daily
• Renewal must stay ahead of SVID rotation, a timing contract the agent has to honour

Choose differently when

If the engine gained native certificate-bound tokens and a generation check, the wrapper should be deleted and the engine's token handed to the agent directly.

Why it holds up over time

The binding applies RFC 8705's idea of certificate-bound tokens at the gateway. The format is versioned and invisible to applications, and every session in existence expires within 12 hours, so a new format can replace the old one in a single release with no migration.

How is a non-production identity kept out of production secrets in a way that no policy mistake can undo?

Context

The requirement asks that cross-environment access be denied structurally, not by any policy a human can write. Policy languages can express anything, including a wildcard that grants staging a production path. The check must happen before policy is consulted and must fail closed when anyone gets it wrong.

Decision

Each environment is its own SPIFFE trust domain under its own offline root. Production gateways trust only the production bundle, so a staging SVID cannot complete a TLS handshake with them. Engines, KMS clusters and upstream admin credentials are per environment. No federation relationship exists between environment trust domains. As defence in depth, the gateway also compares the trust domain with the partition before routing.

How it works on-premise

Three offline roots in HSMs, one per environment, and separate SPIRE deployments. The gateway's client CA pool contains only its environment's bundle. An estate cluster belongs to exactly one environment; clusters that mix environments are not permitted to run the agent.

Options weighed

• ChosenA trust domain per environment: The environment boundary is enforced by certificate path validation, before any request is parsed.
• RejectedOne trust domain with environment in the identity path, enforced by policy: One PKI to run. A single template or wildcard error grants production.
• RejectedOne trust domain with a prefix check in gateway code: Better than policy alone, and kept as a second check. As the only control it is a string comparison one refactor away from being wrong.
• RejectedNetwork separation alone: Cheap where it already exists. Routes change and firewall rules drift; identity has to carry the boundary itself.

Consequences

What it buys

• A policy error cannot reach across environments, because policy is never evaluated
• A staging compromise yields nothing usable in production
• Audit and blast-radius reporting partition naturally by environment

What it costs

• Three PKIs, three sets of ceremonies and three SPIRE deployments
• Mixed-environment clusters must be separated before they can adopt the platform
• Tools that span environments need one identity per environment

Choose differently when

An estate whose non-production and production workloads share clusters cannot adopt this until those clusters are split. In the meantime, a shared trust domain with the gateway prefix check is the interim, recorded as an exception.

Why it holds up over time

It rests on X.509 path validation, the most heavily scrutinised code in any TLS stack. It depends on no policy language, no engine and no product feature, and it stays correct however the platform above it changes.

What is the unit of compromise and failure, and how many copies of the platform does the organisation run?

Context

An OpenBao cluster holds one barrier keyring in memory on its active node; anyone who reads that memory reads everything the cluster stores. Namespaces separate policy, not memory. The requirement asks that one broker compromise not yield another partition's material, that upstream admin credentials never span environments, and that the compromise of storage, key manager or broker memory each yield a bounded and stated subset.

Decision

For each environment at each site, run one KMS cluster, one custody cluster, a set of broker shards divided by upstream family, one gateway pool and one SPIRE deployment. Brokers and custody never share a barrier. The partition map, which says which endpoint serves which environment, site and shape, is data the gateway and agents read.

How it works on-premise

Production at each site: three KMS voters with an HSM pair, five custody voters, two broker shards of five voters, six gateways, three SPIRE servers and a three-node datastore. Staging runs the same shape at both sites on VMs with three voters per cluster; development runs at site A only.

Options weighed

• ChosenSeparate clusters by environment, site and shape: Each memory image holds one environment's one kind of secret.
• RejectedOne global cluster with namespaces per environment: Cheapest to run. One barrier, one unseal, one active node's memory holding every environment.
• RejectedOne cluster per environment with both shapes: Half the clusters. A broker node's memory would hold partner keys alongside upstream admin credentials.
• RejectedOne cluster per team: Strong isolation. Hundreds of clusters, and a boundary that follows the organisation chart, which changes more often than the threat model.

Consequences

What it buys

• The blast-radius statement for each component is short and true
• Upgrades, drills and incidents are staged one partition at a time
• A storm of brokered writes cannot starve custodial reads

What it costs

• About thirty machines per production site, and many clusters to patch and drill
• Tooling must understand the partition map
• Some upstreams need a broker shard of their own to stay within a clean boundary

Choose differently when

An organisation with a few hundred workloads and a single environment that matters should run one cluster per environment and accept a shared barrier. The three-way split costs more than it contains at that size.

Why it holds up over time

Partitions are rows in a map, not code paths. Adding a shard, a site or a new trust tier such as a payment-card enclave is a new row and a new cluster of the same shape, not a redesign.

Where does the platform itself run, given that almost everything in the estate needs it in order to start?

Context

After a site-wide power event, databases, Kubernetes workloads and even the estate's CI need credentials before they can start. If the platform ran on the estate's Kubernetes, it would wait on a cluster whose workloads wait on it, and the estate's node administrators could read the platform's memory.

Decision

KMS, custody, broker, gateway and SPIRE servers, with their datastore, run on dedicated bare-metal hosts in a security enclave: their own racks in three rooms per site, their own VRF, static binaries under systemd, configured by Ansible from signed artefacts already staged on each host. The governance plane runs on a small separate Kubernetes cluster that is an ordinary consumer of the platform.

How it works on-premise

Disks encrypted with LUKS and bound to the TPM, Secure Boot, no interactive shell except through a recorded bastion, nftables allowing only the gateway port from the estate, and time from two independent stratum-1 sources through chrony.

Options weighed

• ChosenDedicated enclave hosts under systemd: Starts with nothing beneath it but hardware, network, time and HSMs.
• RejectedThe estate's Kubernetes: Bin-packing and familiar tooling. A cold-start loop, and estate node administrators with access to platform memory.
• RejectedA dedicated Kubernetes cluster for Tier 0: Good tooling with isolation. Adds etcd, a scheduler and a network plugin to the start-up path, and its own secrets need a bootstrap of their own.
• RejectedVMs on the estate hypervisor: Fast to provision. Hypervisor administrators can read guest memory.

Consequences

What it buys

• A cold start of the whole site has a defined order, and the platform is first
• The set of people with host access to Tier 0 is small and named
• Failure analysis involves no orchestrator

What it costs

• Host management owned by the platform team rather than the estate platform
• No bin-packing; capacity for storms is provisioned in advance
• Slower to add capacity than a pod scale-out

Choose differently when

An organisation that already runs a hardened Tier 0 Kubernetes platform for identity systems, with a tested bootstrap and restricted administration, can reasonably run the engines there.

Why it holds up over time

The rule is testable and independent of technology: nothing Tier 0 needs at start may itself need Tier 0. A yearly cold-start drill checks it whatever runs underneath.

How does the platform survive losing a site without letting one site's failure or compromise reach the other?

Context

The requirement asks for lease state per region, custodial material replicated between regions with an RPO of zero, and no cross-region call on the issuance path. OpenBao's community edition has no cross-cluster replication. A stretched cluster would put inter-site latency on every write and would need a third location to break ties.

Decision

Every cluster is local to its site. Leases and sessions are never replicated: a site outage is survived by the other site's workloads, not by failing leases over. Custodial versions are encrypted separately under each site's own envelope key, written to both custody clusters as staged, and promoted to active only when both writes have committed. Audit replicates asynchronously.

How it works on-premise

The rotation controller and the registry write path hold encrypt-only permission on both sites' KMS clusters. Neither site's KMS key material ever leaves its site. An emergency rotation during a site outage writes to the surviving site only, marks the version single-site, and the returning site is forced to that version before it reopens.

Options weighed

• ChosenSite-local clusters with dual write before promotion: RPO zero for custody by protocol, with independent keys and failure domains.
• RejectedA stretched Raft cluster with a witness site: One cluster to operate. Couples both sites' availability and latency, and needs a third location.
• RejectedAsynchronous primary and replica: Simple writes. A failover can promote a rolled-back version and reintroduce a retired key.
• RejectedReplicating envelope key material between sites: One ciphertext per version. A key present at two sites has two attack surfaces.

Consequences

What it buys

• Sites fail, and are compromised, independently
• No witness location is needed
• Custody RPO is zero without a stretched quorum

What it costs

• Workloads must run at both sites to survive a site loss, which is an estate obligation
• Routine rotation pauses while either site is down
• Two ciphertexts per version to verify and reconcile

Choose differently when

An organisation with a metro pair under two milliseconds apart and a third location may prefer a stretched custody cluster, trading coupling for simpler writes.

Why it holds up over time

The rule that no quorum spans sites can be checked by reading the deployment inventory, and adding a third site is additive: one more write before promotion.

Build a secrets engine, adopt an open-source one, or buy one?

Context

The requirement needs a lease manager with revocation handles, dynamic credential plugins for 14 kinds of upstream, versioned key-value storage, non-exportable cryptographic operations, policy evaluated on every request and audit devices that block on failure. It must be open source and run on premises. HashiCorp Vault moved to the Business Source License in 2023 and no longer qualifies. OpenBao is the Linux Foundation fork of Vault's last open-source release, under MPL 2.0.

Decision

Adopt OpenBao for leases, dynamic credentials, KV v2, transit and policy evaluation. Build the issuance gateway, the credential agent, the registry compiler and applier, the rotation controller, the reconciler, the access broker, and plugins for upstreams that have none. Keep each built component small enough for one engineer to read in an afternoon.

How it works on-premise

A pinned OpenBao release on integrated Raft storage over NVMe. Plugins are built in CI and registered by checksum. Each cluster has one active node and standbys; broker clusters are sharded by upstream family to spread writes.

Options weighed

• ChosenOpenBao with small built edges: Inherits a decade of production use of the Vault lease and plugin model under an open-source licence.
• RejectedHashiCorp Vault Community: The same engine with a larger ecosystem. The Business Source License fails the open-source constraint.
• RejectedA new engine built on PostgreSQL: Full control over scaling. Years of security-critical code to rebuild leases, plugins and sealing.
• RejectedCyberArk Conjur Open Source: A sound policy model for custody. No dynamic credential brokering comparable to database plugins.
• RejectedKubernetes Secrets with External Secrets Operator: Native to the platform. Secrets land in etcd and usually in environment variables, with no leases.

Consequences

What it buys

• Leases, revocation, plugins and blocking audit devices are mature and understood
• Policy is evaluated per request against the current policy set
• The built code is limited to the requirements no engine enforces

What it costs

• One active writer per cluster, which forces sharding and a proven write ceiling
• A younger project community than Vault's, whose security release cadence must be watched
• Built plugins are tied to OpenBao's plugin interface

Choose differently when

If the proof shows a single shard cannot sustain 1,000 lease writes a second at the target latency, keep OpenBao for custody and transit and move lease state to a horizontally scalable store behind the gateway.

Why it holds up over time

No application sees OpenBao's API; the agent protocol is the contract. OpenBao can give way to a relicensed Vault, a successor fork or a built engine behind the gateway without a single workload changing.

Where do the checks live that the engine cannot express, and is another hop worth it?

Context

OpenBao cannot bind a token to a client key, evaluate environment before policy, enforce a restore generation, put renewals ahead of first issuance, apply a different audit posture to different event classes, or require a second identity to decrypt custodial material. Each is a stated requirement. Without a component that owns them they would be scattered across plugins, agent code and conventions.

Decision

Build a Go gateway, stateless and horizontally scaled. It terminates mTLS against the environment bundle, validates the JWT-SVID against the connection, wraps and checks sessions, enforces the generation, rate-limits per identity and per secret, runs two admission lanes, routes to engines by the partition map, unwraps custodial ciphertext through the KMS, and writes synchronous audit records for custodial and human events.

How it works on-premise

Six gateways per production site, twelve during a planned storm. Agents hold a static, signed endpoint list per partition and balance on the client side, so no load balancer and no DNS sits on the issuance path.

Options weighed

• ChosenA built gateway: Every non-negotiable check in one reviewed place of a few thousand lines.
• RejectedEnvoy with external authorisation and scripted filters: Good at mTLS and rate limits. Binding, generation, unwrap and synchronous audit would still be custom code, now split across filter languages.
• RejectedOpenBao plugins only: No new hop. Plugins cannot see the original client's TLS connection and cannot reorder admission.
• RejectedNo gateway, accepting the gaps: The simplest diagram. Fails key binding, per-class audit posture and renewal priority.

Consequences

What it buys

• The engine can be replaced without re-implementing the requirements
• Capacity is added by adding instances, with no state to move
• One place to fuzz, review and test the security-critical checks

What it costs

• About two milliseconds of added latency
• Security-critical code owned by the platform team
• One more component to release and upgrade

Choose differently when

If the engine gains key-bound tokens and per-path audit posture, the gateway shrinks to rate limiting and admission and could become configuration on an off-the-shelf proxy.

Why it holds up over time

The gateway's responsibilities are a list derived from requirements, not from a product's permanent gaps. Each can be deleted on its own when an engine provides it, so the gateway gets smaller with time rather than accreting.

What is the root of trust for material at rest, and who controls it?

Context

The requirement asks for a key manager the platform does not control, with independent access control, its own audit of every key use, and the property that withdrawing access leaves the platform unable to decrypt anything. There is no cloud KMS on premises. HSMs are the right place for a root key but make poor per-read envelope services: identity-aware policy and rich audit would have to be rebuilt around them.

Decision

For each environment at each site, run a three-node OpenBao cluster used only for transit keys and as the auto-unseal for that partition's custody and broker clusters. Seal it with an HSM pair over PKCS#11. The key custodians in the security office operate it; the platform team holds no policy on it. Its audit devices ship to the records team. Withdrawing the gateway's decrypt policy stops custodial reads on the next request; sealing the engines, a command custodians hold, stops everything else at once.

How it works on-premise

A pair of FIPS 140-3 Level 3 network HSMs per site. Recovery keys split three of five on hardware tokens held by named custodians. KMS hosts in their own racks. The engines authenticate to the KMS for auto-unseal with a narrowly scoped periodic token stored on TPM-bound encrypted disks; that token is the platform's own first secret-zero exception and is registered as such.

Options weighed

• ChosenOpenBao KMS on an HSM seal, run by custodians: Identity-aware policy and audit in software, the root key in hardware, and a different team at the controls.
• RejectedHSMs called directly as the KMS: Fewer layers. Per-read, per-identity policy and structured audit would be rebuilt around a PKCS#11 session.
• DeferredA KMS cluster unsealed by custodians with Shamir shares: No HSM dependency. Every KMS restart needs custodians present; kept as the fallback if the HSM seal fails its proof.
• RejectedThe platform team runs the KMS too: Simpler rota. Independence becomes a claim rather than a fact.

Consequences

What it buys

• The key manager's log is independent of the platform's own
• Access withdrawal is real and exercised in drills
• Root key material never exists outside an HSM-sealed barrier

What it costs

• HSM hardware, support contracts and a custodian rota
• Another cluster per partition on the critical path of custodial reads
• An unseal token on engine hosts, written down as an exception rather than hidden

Choose differently when

If the organisation already runs an enterprise key management service with separate administration and its own audit, use its HSMs as the seal root and keep the OpenBao KMS layer for identity-aware envelope operations.

Why it holds up over time

Envelope keys are versioned. Moving to new HSMs, larger keys or a post-quantum algorithm is a key rotation followed by an in-place rewrap of ciphertext, which never exposes plaintext to storage and never touches a workload.

What does possession of the custody store yield, and what should a KMS outage do to custodial reads?

Context

OpenBao's barrier encrypts storage, but custody nodes hold the decrypted barrier key in memory, so reading one node's memory reads every secret. The requirement asks for every key use to appear in the key manager's own log, for custodial reads to fail closed when the key manager is down, and for storage, key manager and engine memory each to yield a bounded subset when compromised.

Decision

Custody stores only ciphertext produced by the KMS under a per-site envelope key. The workload's policy authorises reading the ciphertext; only the gateway's identity may decrypt it. Decryption happens on every read, and no decrypted copy is cached anywhere in the enclave. Ciphertext is bound to its secret path with a derived-key context, so ciphertext copied from one secret to another will not decrypt.

How it works on-premise

Transit keys with derivation enabled and the secret path as context. One envelope key per environment per site, rotated yearly with lazy rewrap on read and a background rewrap job. Versions are capped at 64 KB.

Options weighed

• ChosenEnvelope per version, unwrapped per read: Needs two identities and two clusters to produce plaintext, and leaves a complete independent record.
• RejectedThe OpenBao barrier alone: No KMS on the read path. One custody node's memory yields every custodial secret.
• RejectedUnwrap once and cache plaintext in the gateway: Faster, and survives a KMS blip. The KMS log stops being complete and an outage stops failing closed.
• RejectedDecrypt in the agent: Removes a hop. Every node needs KMS access, which multiplies the KMS's attack surface by two thousand.

Consequences

What it buys

• Storage, custody and KMS compromises each yield a stated, smaller subset
• The KMS log is a complete read history the platform team cannot edit
• Ciphertext cannot be swapped between secrets

What it costs

• The KMS sits on the custodial read path, adding two to five milliseconds and an availability dependency by design
• The KMS must be sized for custodial read volume
• Renaming a secret requires a rewrap

Choose differently when

If custodial reads grew into thousands a second, cache per-version data keys, never plaintext, in gateway memory for a few minutes, and accept a coarser KMS log.

Why it holds up over time

Envelope encryption with an external key manager is the construction every commercial KMS implements. Nothing about it depends on OpenBao, and any future custody engine can store the same ciphertext.

Broker every credential and refuse to store anything, or accept a custodial tier, and how are the two kept from being confused?

Context

Partner API keys, signing keys and legacy accounts cannot be minted, and they are often the most consequential secrets in the estate. A platform that refuses them does not make them go away; they go back into CI variables. A platform that accepts them reintroduces long-lived material and a store worth stealing. The danger in having both is that people assume the custodial secret expires like the brokered one.

Decision

Accept a custodial tier as an admitted compromise with its own guarantee. Each secret's shape is declared in the registry and never inferred. Brokered roles promise a lifetime and revocation. Custodial secrets promise custody, controlled release, rotation and a maximum age. They live in different clusters, under different paths, with separate dashboards and SLOs. The agent presents one interface, so a secret that moves from custody to brokered changes no application code. The inventory marks each custodial secret whose upstream could support minting, as a migration candidate.

How it works on-premise

Brokered roles under database/, rabbitmq/ and built plugin mounts on broker shards; custodial secrets under kv/ and transit/ on custody clusters. The registry schema makes shape a required field and rejects a custodial secret without an owner, a rotation posture, a maximum age and a consumer set.

Options weighed

• ChosenTwo declared products: Each guarantee is stated where it is true and nowhere else.
• RejectedBroker only, refuse custody: One guarantee and one lifecycle. The refused secrets return to wherever they came from.
• RejectedCustody only, with scheduled rotation: Simple and familiar. Static credentials with better hygiene, and no per-instance accounts.
• RejectedOne engine, shape inferred from the upstream: Less to declare. Inference is wrong often enough to make every guarantee doubtful.

Consequences

What it buys

• Nobody is told a partner key expires when it does not
• Each tier is sized and monitored for its own failure modes
• The route from custody to brokerage is visible and needs no application work

What it costs

• Two runbooks and two sets of SLOs
• People will still call both of them the vault
• The custodial tier remains a store worth attacking

Choose differently when

A greenfield estate whose upstreams can all mint accounts or accept federation should refuse the custodial tier and handle the few exceptions through the register.

Why it holds up over time

The distinction is a property of upstreams, not of tools. As upstreams add account creation or federation the custodial list shrinks, and nothing in the design changes.

How does a workload use a signing or decryption key without ever holding it?

Context

Signing keys copied to build agents are among the most damaging leaks an organisation can have, because what they sign is trusted for years. The requirement asks that a secret declared non-exportable be exposed only as an operation.

Decision

A non-exportable secret is generated in, or imported into, the custody cluster's transit engine with export disabled. Workloads call sign, verify, encrypt or decrypt through the gateway under per-identity policy that separates signing from verifying. Break-glass access cannot read these keys either.

How it works on-premise

Transit key types chosen per consumer: Ed25519, ECDSA P-256, RSA-3072 and AES-256-GCM. Existing keys are imported with transit's wrapped import. Signing tools that support a Vault-compatible KMS provider, such as cosign, use it directly; the rest use a small adapter. Operations are rate-limited per key.

Options weighed

• ChosenTransit operations through the gateway: The key never leaves the barrier, and every signature has a named caller.
• RejectedPKCS#11 access to the HSM per workload: The strongest key protection. HSM partitions and client libraries sprawl across every consuming team.
• RejectedExport to the workload with memory protections: No latency. The key then lives in the memory of every pod that uses it.
• RejectedA signing service per team: Local control. The same service built, reviewed and got wrong many times.

Consequences

What it buys

• A compromised build agent can sign while it is compromised, and not afterwards
• Every signature and decryption is attributable
• Key rotation is invisible to callers

What it costs

• A network call per operation, and availability coupling for signing workloads
• Some libraries expect a key file and need an adapter
• Bulk decryption at high rates is not a good fit

Choose differently when

For high-volume data decryption inside a workload, issue per-object data keys wrapped by transit, and keep only the wrapping key non-exportable.

Why it holds up over time

An interface that says sign this is stable across algorithms. Moving to post-quantum signatures changes a key type and the verifiers, not the code path of every signer.

How short is short, and who decides?

Context

A fifteen-minute lease contains a leak to fifteen minutes only if nobody renews it. While the legitimate holder is alive and renewing, a leaked copy lives as long as the total renewable lifetime. Shorter leases add renewal load, account churn and audit volume, and shorten how long workloads survive a control-plane outage. The requirement sets defaults of 30 minutes, 60 at most and 12 hours in total, and 15 minutes with no renewal for irrevocable roles.

Decision

The platform bounds are a 30-minute default, a 60-minute maximum and a 12-hour total, with 15 minutes and no renewal for irrevocable roles. Each role declares its own default, maximum and total within those bounds; owners of high-value roles set a shorter total. A caller may request less than the role's default and never more. The registry pull request shows the renewal rate, account churn and audit volume the chosen values imply.

How it works on-premise

OpenBao role default and maximum TTLs; the total lifetime is enforced by the session's 12-hour maximum and by the agent re-minting at the total. The compiler rejects out-of-bounds values and computes the load delta from the lifetime model on view 28.

Options weighed

• ChosenPlatform bounds, per-role values, shorten-only callers: The trade is made by the people who understand it, with the cost in front of them.
• RejectedOne fixed platform lifetime: Nothing to configure. A database role and an irrevocable token endpoint need different numbers.
• RejectedCallers choose within a range: Flexible. Every caller chooses the maximum.
• RejectedLong leases with strong revocation: Low load. Containment then depends on revocation, which is the exception path.

Consequences

What it buys

• Exposure is explained separately for a dead holder and a live one
• High-value roles can buy shorter exposure at a visible price
• Load is predictable from declared values

What it costs

• Variety in configuration that operators must understand
• Renewal load grows as owners shorten lifetimes
• The live-holder case still needs revocation and detection

Choose differently when

If leak detection proves precise, total lifetimes can lengthen for low-value roles. If an upstream accepts federation, the question disappears for that upstream.

Why it holds up over time

The model is arithmetic: renewals are outstanding leases divided by two-thirds of the lifetime. It holds for any engine, and the published table is regenerated from telemetry rather than argued from memory.

What revokes a pod's credentials when the pod disappears and nobody tells the platform?

Context

Pods are killed without warning, nodes lose power, and deletion events from Kubernetes are sometimes missed. The requirement asks for revocation when the identity a credential was issued to disappears. Any design that waits for a message saying so will sometimes wait forever.

Decision

Every lease is created under the session's OpenBao service token, and OpenBao revokes a token's leases when the token expires or is revoked. The agent renews a pod's session only while that pod's volume is mounted on its node. A pod that vanishes stops being renewed, and within 15 minutes its session and every lease under it are revoked. A clean unmount revokes immediately as a fast path.

How it works on-premise

Service tokens with a 15-minute TTL, renewable to 12 hours, one per pod instance; about 70 token renewals a second across the estate. Revocation of child leases runs in OpenBao's expiration manager and is confirmed at upstreams by the reconciler.

Options weighed

• ChosenA lease tree under a renewed session: Silence produces containment; no message is needed.
• RejectedA central watcher on Kubernetes pod deletions: Immediate when it works. Misses events and depends on API server availability; kept only as the local unmount fast path.
• RejectedLease TTL alone: No coupling to sessions. Leases outlive the pods they belong to by up to their full lifetime.
• RejectedAn exit hook in a sidecar: Precise on graceful shutdown. A kill signal or node loss never runs it.

Consequences

What it buys

• Node loss and pod loss are handled by the same mechanism
• Revoking one session contains everything issued through it
• Correctness does not depend on any event arriving

What it costs

• A window of up to 15 minutes after an unannounced disappearance
• Session revocation cascades, so a bug that revokes sessions breaks credentials broadly
• About 70 token writes a second across the estate

Choose differently when

If agents were ever replaced by application SDKs, those SDKs would have to renew sessions under the same contract, or this property would be lost silently.

Why it holds up over time

It is the dead-man's-switch pattern: the default outcome of silence is safety. The pattern survives any change of engine, orchestrator or agent that preserves the contract.

How is the credential that creates credentials kept from being the largest leak in the system?

Context

A broker must hold an account on each upstream that can create and delete accounts. That account is more valuable than everything it creates. The requirement calls these the crown jewels, asks for the tightest custody and shortest rotation, and forbids any one of them covering more than one environment.

Decision

Each broker connection uses its own admin account per upstream cluster per environment, with privileges to create, alter and drop platform-named accounts and to grant membership in pre-created privilege roles, and nothing else. On PostgreSQL 16 and later, an account with CREATEROLE manages only the roles it created. OpenBao rotates each admin password daily so no human knows it, and no read path returns connection configuration. The reconciler uses a separate account that can only list and drop platform-named accounts.

How it works on-premise

Privilege roles such as orders_rw are created by DBAs and granted to minted accounts; minted accounts carry VALID UNTIL and a connection limit. RabbitMQ admin users are limited to the platform's virtual hosts. A failed root rotation is recovered by a DBA pair re-seeding the account through break-glass.

Options weighed

• ChosenScoped admin per upstream per environment, rotated daily: A broker compromise yields account-creation rights, not data.
• RejectedA superuser per upstream: No DBA set-up. A broker compromise becomes a data breach.
• RejectedAdmin credentials held in custody and fetched by the broker: One store for everything. Adds a read path to the most valuable credentials.
• RejectedOne admin across environments for shared database fleets: Fewer accounts. Explicitly forbidden by the requirement, and rightly.

Consequences

What it buys

• Nobody, including platform operators, knows any admin password
• Compromise of one shard's memory is bounded to one environment's upstream family
• Orphan cleanup cannot create anything

What it costs

• DBA work to set up scoped admin and privilege roles on every cluster
• PostgreSQL 16 or later required for the tight CREATEROLE scoping
• A failed root rotation needs a two-person recovery

Choose differently when

For upstreams with no scoped administrative model, give them a broker shard of their own so their admin credential shares a memory image with nothing else, or keep them custodial.

Why it holds up over time

Least privilege for machine administrators, and a separate account for cleanup, are properties of the upstreams and the registry. They carry over unchanged to any engine.

What does revoked mean for an upstream that cannot revoke?

Context

Some internal token endpoints issue signed tokens with no revocation list, and some legacy services create keys they cannot delete early. The requirement's options are to refuse them, to broker with leases short enough that expiry is the containment, or to hold a credential custodially and rotate it after an incident.

Decision

Broker them, with a maximum of 15 minutes and no renewal, and mark the role irrevocable. Each such role has an exception register entry with an owner and a review date. Revoking an irrevocable lease stops re-issuance and is reported as expiring at a stated time, never as revoked.

How it works on-premise

Built plugins for internal token endpoints declare their revocation method; the compiler refuses a plugin with none unless the role carries the irrevocable flag. Agents re-mint at about ten minutes, and the upstream's churn budget is checked in the registry pull request.

Options weighed

• ChosenShort, unrenewable brokered leases: The containment bound is exactly the lease, and it is written down.
• RejectedRefuse to broker: Principled. Owners fall back to static tokens, which is worse.
• RejectedHold a custodial credential and rotate on incident: No upstream churn. Containment waits for someone to notice; allowed only where churn cannot be absorbed, and registered.
• Right elsewherePut token introspection in front of the upstream: The real fix, owned by the upstream's team. The register entry is how that work gets asked for.

Consequences

What it buys

• An honest, known bound on every irrevocable credential
• Irrevocable upstreams are visible on the register rather than hidden in a success rate
• Owners have a reason to add revocation

What it costs

• Continuous re-minting load on the upstream
• A leaked irrevocable credential is usable for up to 15 minutes regardless of response
• Brief interruptions if minting is unavailable for longer than a lease

Choose differently when

If an upstream cannot tolerate the mint rate that 15-minute leases imply, custody with rotation is the lesser evil, registered with its slower containment stated.

Why it holds up over time

Entries close as upstreams add revocation or federation, and the class shrinks without any design change.

What stops the platform from taking down an upstream it exists to serve?

Context

A full node-pool replacement asks for hundreds of thousands of new accounts in minutes. PostgreSQL creates roles at a few hundred a second per primary, and every minted credential that connects consumes a connection slot. Seven hundred roles share fourteen kinds of upstream. The requirement asks for caps per role, shedding before exhaustion, and partitioning so one high-churn role cannot starve others on the same upstream.

Decision

Each role declares a lease cap, by default three times its expected instance count. Each upstream declares a mint budget per second and a connection budget. The gateway enforces both: per-role concurrency from the broker's outstanding-lease metric, refreshed every ten seconds and applied at 90% of the cap, and a token bucket per upstream. Requests over budget receive a retryable 429 naming the upstream. The renewal lane is never subject to mint budgets.

How it works on-premise

Mint budgets start at 200 role creations a second per PostgreSQL primary and are tuned from the proof phase. The OpenBao database plugin's connection pool is capped per upstream. Minted PostgreSQL roles carry a connection limit set by the role declaration.

Options weighed

• ChosenCaps and budgets enforced at the gateway: Sees every role on a shared upstream, and sheds before the upstream feels it.
• RejectedEngine rate-limit quotas alone: No built code. Quotas are per path and cannot see several roles sharing one upstream.
• RejectedLet upstream errors push back: Nothing to configure. The upstream is already hurting by the time it errors, and so is every other tenant on it.
• DeferredA broker shard per high-churn upstream: Physical isolation. Used when a single upstream dominates a shard's write load.

Consequences

What it buys

• A restart storm slows new starts instead of breaking running databases
• One noisy role cannot exhaust an upstream shared with others
• Budgets are declared data that the upstream's owner can review

What it costs

• Budgets need tuning per upstream and drift as fleets grow
• Metric-based caps are approximate within a ten-second window
• New pods of a capped role wait during storms

Choose differently when

Upstreams that federate workload identity have no accounts to create, and their budgets can be removed.

Why it holds up over time

Admission control that protects a shared dependency is independent of the engine. The budgets are data in the registry and move with the upstream, not with the platform.

What runs next to the workload to obtain, renew and deliver its credentials, and how many copies of it are there?

Context

The requirement asks for a local agent so that applications never handle attestation, renewal or the platform's address, never write to disk, never extend a lifetime, notify on change, and continue from memory during an outage. The existing choices are OpenBao Agent or Proxy as a sidecar in every pod, the Secrets Store CSI driver with a provider, or client libraries in each language.

Decision

Build one Go agent per node, deployed as a DaemonSet. It is a CSI node plugin that mounts a per-pod socket directory and a per-pod tmpfs volume; a registered delegate of the SPIRE agent, obtaining each local pod's SVIDs through the Delegated Identity API; the session and lease manager; an SDS server for proxy mode; and the enforcer of monotonic ageing, the skew guard, jittered attestation and change notification.

How it works on-premise

Pages locked with mlock and the process marked non-dumpable. Change notification through a watch stream on the socket and atomic rename for files. Rolled out through Argo CD to 1%, 10% and then all nodes, with the gateway supporting the current and previous agent protocol.

Options weighed

• ChosenA built agent per node: One process per node, one implementation of every invariant, and a blast radius equal to what a node compromise already yields.
• RejectedOpenBao Agent sidecar per pod: Mature and adopted. About 40,000 processes and some 800 GB of memory, no delegated identity, per-application configuration, and templating that invites environment variables.
• RejectedSecrets Store CSI driver with an OpenBao provider: Standard and unobtrusive. File delivery at mount time with polled rotation, no socket and no lease-aware renewal; right for a file-only estate.
• RejectedClient libraries in each language: No daemon. Every language, every team and every bug fixed many times over.

Consequences

What it buys

• The invariants are enforced by one program on every node
• Per-node blast radius is enumerable and matches the node's own
• Applications code against a local socket and nothing else

What it costs

• A security-critical daemon on every node, owned by the platform team
• Correctness depends on the SPIRE agent's attestation of local pods
• A defect affects every pod on a node, which demands staged rollout

Choose differently when

An estate of a few hundred pods should use OpenBao Agent sidecars; the cost of building and owning a node agent is not repaid at that size.

Why it holds up over time

The socket protocol is the contract applications depend on, so it is the one interface to keep backwards compatible. Gateways, engines and SPIRE can all change behind it.

Where does the credential land: an endpoint the application calls, a file it reads, or a proxy that uses it on the application's behalf?

Context

An endpoint is rotation-native but requires application change. A file works unmodified but is readable by anything in the container and makes rotation a file-watch problem. A proxy gives the strongest containment but only for protocols it can speak. Environment variables are the most common choice in practice and the worst: inherited by every child, printed in crash dumps and frozen until restart.

Decision

Offer three modes permanently. The local socket is the default. A tmpfs file serves software that cannot change, and such workloads are registered as unmodifiable consumers and excluded from routine short-lease rotation until they change. An Envoy credential injector serves HTTP APIs whose key the application should never see. Database wire protocols are not proxied. Environment variables are refused: the agent has no such mode, and admission policy rejects pods that reference platform paths through the environment.

How it works on-premise

The socket speaks HTTP over a unix domain socket with a watch endpoint. Files are written by atomic rename, 0400, owned by the pod's UID, removed on unmount. Envoy receives credentials over SDS from the agent. Admission rules run in the estate's existing admission controller.

Options weighed

• ChosenSocket default, file and proxy by declaration, no environment variables: Every workload has a supported path, and the unsafe one is closed.
• RejectedSocket only: One path to secure. Unmodifiable software has nowhere to go and returns to static credentials.
• RejectedFile only: Works everywhere. Readable by every process in the container and rotated by file watching.
• DeferredA database proxy holding credentials: Keeps passwords out of applications. Pooling changes transaction and session semantics; evaluated per team.
• RejectedEnvironment variables: What every framework expects. Readable by children, captured in dumps and unrotatable without a restart.

Consequences

What it buys

• The safest mode is the easiest one
• Legacy workloads are supported and named, not ignored
• An application in proxy mode cannot leak a key it never had

What it costs

• Three delivery paths to test and support indefinitely
• Framework integrations must be written for the socket
• Teams used to environment variables need migration help

Choose differently when

An estate made almost entirely of unmodifiable vendor software would make file delivery the default and invest in consumer-side reload hooks.

Why it holds up over time

Each mode uses a primitive older than the platform: unix sockets, tmpfs and SDS. They will outlast any agent implementation.

What should agents do when the control plane is unreachable?

Context

The options are to serve cached credentials until they expire, to extend them for a bounded grace period, or to fail closed at once. Serving to expiry turns a long outage into rolling estate-wide failure. A grace period prevents that, but at the moment of decision an outage and a compromise look the same, and every containment claim becomes conditional on nobody having used the grace. Failing closed at once is defensible only if the platform is more available than everything that depends on it.

Decision

Agents serve valid cached credentials until their own expiry and not a moment longer. No agent setting, gateway flag or operator command extends a lifetime. Custodial copies are evicted 60 minutes after they were read. Availability is bought through the control plane: 99.99% issuance per site, five voters in three rooms, a 10-minute lease-store RTO and quarterly outage drills.

How it works on-premise

The agent's cache has no code path that alters an expiry, and the invariant suite runs every failure mode against it and fails the build if any credential is served past its deadline.

Options weighed

• ChosenServe until expiry, no extension: Every stated lifetime remains the real lifetime.
• RejectedA bounded emergency grace: Rides out long outages. Makes every lifetime conditional and puts a dangerous switch in the hands of whoever is on call.
• RejectedFail closed immediately: The strictest posture. Turns every network blip into an estate outage.
• RejectedLonger leases for resilience: Survives longer outages. Trades containment for availability permanently rather than during an outage.

Consequences

What it buys

• Containment claims hold during outages as well as in steady state
• No one can be pressured into extending lifetimes during an incident
• The outage profile is known in advance and drilled

What it costs

• After 60 minutes of site outage, credential-dependent workloads at that site fail
• The platform's availability target becomes the estate's ceiling
• The degradation must be explained to every service owner

Choose differently when

A workload whose own availability target exceeds the platform's should not depend on brokered credentials. It should use federation where the upstream supports it, or a registered custodial credential with a long maximum age.

Why it holds up over time

The absence of a switch is the most durable control there is. No future incident commander can flip what was never built, whatever the pressure.

How does a node with a wrong clock avoid serving an expired credential or refusing a valid one?

Context

Wall clocks jump: NTP corrections, virtual machine pauses, a mis-set BIOS after hardware replacement. An agent that compares an absolute expiry with a wrong clock serves credentials that have expired or discards ones that have not. The requirement sets a ±30 s tolerance and asks that an agent beyond it refuse to serve.

Decision

Every credential is issued with a TTL and an absolute expiry. The agent computes its deadline as the monotonic time of receipt plus the TTL minus the round-trip time, and uses that to decide whether to serve. The wall clock is used only for file-mode consumers that read an absolute expiry and to compare with gateway time and chrony status. Beyond ±30 s of skew the agent stops serving, reports it, and the node is cordoned.

How it works on-premise

chrony on every node against the site's stratum-1 sources, skew exported as a metric, and a node controller that cordons on the agent's report.

Options weighed

• ChosenMonotonic ageing with a skew guard: Correct whatever the wall clock does, and loud when the wall clock is wrong.
• RejectedTrust the wall clock: Simplest. Wrong exactly when clocks are wrong.
• RejectedMonotonic ageing with no skew check: Robust in the agent. File consumers and upstream VALID UNTIL still use wall time and would disagree silently.
• RejectedExpiry checked only by the server: One clock. The agent serves from memory when the server is unreachable.

Consequences

What it buys

• Expiry is enforced correctly on nodes with bad clocks
• Clock faults surface as named node problems
• The agent and the upstream agree on when a credential ends

What it costs

• A node with a bad clock loses credential service until fixed
• Another metric and controller to operate
• Round-trip estimation makes deadlines slightly conservative

Choose differently when

None likely. The alternatives are strictly worse, and the only variable is the tolerance, which can be tightened if clock quality allows.

Why it holds up over time

Monotonic clocks exist on every operating system the agent will ever run on, and the rule does not depend on any product.

Who owns the last mile of rotation, and how long may a straggler hold the old credential?

Context

Pushing a new credential into consumers requires reaching into every one of them. Waiting for observed use to stop is safe but unbounded, because one straggler keeps the old credential alive indefinitely. Retiring on schedule gives the window meaning and guarantees breakage the day a consumer is missed. The requirement sets a 24-hour overlap and retirement after 24 hours with no observed use.

Decision

Automated custodial rotation keeps two working credentials: alternating accounts for databases, two keys for APIs that allow it. The new version is staged at both sites, verified with a real call, and promoted. The old version is retired after 24 hours with no observed use. If it is still used on day seven, it is retired anyway and the named straggler breaks, after notices at 24 hours and five days. A secret may declare retire-on-schedule instead. Emergency rotation skips the overlap entirely.

How it works on-premise

The rotation controller in Go, with one rotator per upstream kind. Observed use combines agents' reports of the version each workload holds with the upstream's own evidence: login logs per account, or a partner's usage report per key. A rotation that fails verification aborts and leaves the old version active.

Options weighed

• ChosenPull, observed-use retirement, seven-day ceiling: Safe for well-behaved consumers and bounded for the rest.
• RejectedPush and confirm: Fast and explicit. The platform would need to reach into, and know the health of, every consumer.
• RejectedObserved use with no ceiling: Nobody breaks. One forgotten job keeps the old credential alive forever.
• RejectedRetire on schedule after 24 hours: The window is exact. Every monthly batch job breaks on its next run.

Consequences

What it buys

• Rotations never leave zero working credentials
• Stragglers are found by name, including consumers outside the agent
• The ceiling gives the overlap a meaning without guaranteeing breakage

What it costs

• Upstreams must support two concurrent credentials
• A straggler will break on day seven, deliberately
• Observed use depends on upstream logs being available

Choose differently when

Where a partner allows only one active key, rotation is manual with a named owner, and every rotation is effectively an emergency one.

Why it holds up over time

Dual-credential rotation is a property of how upstreams authenticate, not of the platform. The ceiling is a number in the registry that can move as consumers improve.

How does a responder revoke exactly the right set of credentials, and when may the platform say the job is done?

Context

Incidents rarely involve a single credential. The requirement asks for revocation by lease, identity, role, node, namespace and everything, propagated to upstreams within 30 s at p99 and within five minutes at worst, and for failure to be reported as a live, uncontained credential rather than as success.

Decision

Each scope resolves through the inventory. An identity scope disables its SPIRE entry and revokes all its sessions, taking their leases with them. A node scope evicts the node's SPIRE agent and revokes sessions issued through it. A role scope revokes the role's lease prefix and disables the role. A namespace scope is the union of its identities. Everything closes the partition at the gateway and revokes every prefix. New issuance is stopped before live credentials are revoked. A lease is contained only when the reconciler reads the upstream and finds the account gone; after five minutes it is reported uncontained and pages until closed.

How it works on-premise

Revocation is synchronous in OpenBao. PostgreSQL revocation statements set NOLOGIN, terminate the role's backends, then drop the role. The inventory previews the workloads in scope before execution.

Options weighed

• ChosenScope resolution with upstream confirmation: Stops refilling, then empties, then verifies.
• RejectedTrust the engine's revocation result: Immediate and simple. An API success is not an absent account.
• Right elsewhereRotate the upstream admin and drop every platform account: Right for a compromised broker shard. As a routine tool it is an estate outage.

Consequences

What it buys

• Responders revoke by the question they are asking, not by lease ID
• The report says what was observed, not what was requested
• Unaffected workloads re-mint within seconds

What it costs

• Revocation depends on inventory freshness for scope resolution
• An upstream that is down leaves leases reported as uncontained until it returns
• Wide scopes create re-mint bursts that the admission lanes must absorb

Choose differently when

Upstreams that accept workload identity directly need no account revocation at all; disabling the identity is the whole revocation.

Why it holds up over time

Stop new, end live, verify at the source is the pattern for every revocation system. The scopes are data in the inventory and do not depend on the engine.

How does the platform find a credential that is live at an upstream and unknown to it, or a lease that refers to an account that no longer exists?

Context

The lease store is the platform's record of what is outstanding, but a crashed plugin call, a restored snapshot or a DBA's manual change can make it wrong. A lost lease record is a live account nobody will revoke, which the requirement calls the platform's worst data-loss outcome. It asks for reconciliation at least hourly and an orphan lifetime of at most 90 minutes.

Decision

A reconciler per broker shard lists platform-named accounts at each upstream every 60 minutes, staggered, and compares them with outstanding leases. An account with no lease is revoked through the reconciler's own list-and-drop account. A lease with no account is closed. Accounts without the platform prefix are reported and never touched.

How it works on-premise

Minted usernames carry a fixed prefix and the lease hash, which makes the comparison a set difference. Results go to the audit stream and a containment dashboard; a sudden rise in orphans is itself alerted on.

Options weighed

• ChosenHourly reconciliation against the upstream: Neither store is believed alone, and every disagreement is a finding.
• RejectedTrust the lease store: No upstream reads. Blind to exactly the failures that matter.
• RejectedTrust the upstream: The account list is real. It cannot say which accounts were intended.
• DeferredChange data capture from upstream catalogues: Near real time. Heavy integration for a bound that hourly polling already meets.

Consequences

What it buys

• Orphans have a hard maximum lifetime
• Restores and plugin crashes are corrected without human intervention
• Manual account changes by others become visible

What it costs

• Hourly list queries against every upstream
• Up to 90 minutes of orphan life
• Reconciler accounts to scope, rotate and audit

Choose differently when

If an upstream can tag accounts with the lease that created them and expire them itself, reconciliation for that upstream becomes a consistency check rather than a cleanup.

Why it holds up over time

Reconciling two systems of record is the oldest control in operations, and it works with any engine and any upstream that can list its accounts.

How is a store restored from backup without reviving a revoked credential, an attacker's session or a rotated-out secret?

Context

A snapshot is a copy of the platform's past decisions. Restoring one brings back sessions that were revoked, leases whose accounts were dropped and custodial versions that were rotated out. The requirement asks that restored leases be revalidated against upstreams and that a restore advance a generation so pre-restore sessions are not honoured.

Decision

Each partition has a generation stored in the KMS cluster, which is never restored together with the engines it protects. Restoring a broker or custody cluster requires an SRE and a key custodian to advance the generation. Gateways refuse every session from an earlier generation and keep the partition closed until the reconciler has closed leases with no account, revoked accounts with no lease, and compared custodial versions with the rotation ledger held in the audit store. Secrets whose restored version is older than the ledger says are quarantined until rotated forward.

How it works on-premise

Raft snapshots every 15 minutes to the site's backup store, encrypted by the barrier. The restore runbook is a script with two approvals. It is drilled quarterly in staging against a snapshot taken 40 minutes before a burst of revocations.

Options weighed

• ChosenExternal generation plus revalidation: The past is restored as data and overruled as authority.
• RejectedRestore without any generation: Quick. Revives sessions and leases that were deliberately ended.
• Right elsewhereRebuild brokers empty instead of restoring: Attractive, because leases can be re-minted. It drops every account at once, which is an estate outage; kept as the fallback when a snapshot itself is suspect.
• RejectedNever restore; rebuild everything from the registry: No stale state. Custodial material cannot be rebuilt from intent.

Consequences

What it buys

• No session from before an incident survives a recovery
• Revoked accounts and retired versions cannot return to service
• Restores are safe to rehearse

What it costs

• Every workload in the partition re-attests after a restore
• Revalidation lengthens RTO and must be proven inside ten minutes
• Two people are needed to do something that is technically quick

Choose differently when

A platform whose broker state is small enough to rebuild empty within the RTO, and whose upstreams tolerate mass re-creation, can prefer rebuilding to restoring.

Why it holds up over time

Generation counters and revalidation after restore are how consensus systems and certificate authorities have handled the same problem for decades. The generation's location outside the restored store is the part that must never change.

How are grants made, reviewed and reverted so that the policy in force is exactly what people approved?

Context

The requirement forbids inheritance, wildcards at the top of a path and default grants; asks for a second approval for production custodial grants; asks for policy to be versioned, validated against known-good requests and revertible in one operation; and requires acknowledgement before any change that would deny a currently succeeding production identity. Hand-written engine policy, however carefully reviewed, is reviewed in the wrong language.

Decision

Teams declare identities, secrets, roles and grants in the registry. The compiler rejects wildcards of any kind, grants without an owner and environment mismatches; emits OpenBao policies, JWT roles and SPIRE entries; replays seven days of real requests from the audit store against the result; and computes each identity's reachable set. Production custodial grants and changes that deny current traffic need two approvers, neither of them the author. A single applier identity per environment writes the compiled bundle. Revert is a Git revert through the same pipeline with approval waived.

How it works on-premise

Forgejo with protected branches and code owners, Forgejo Actions for compilation and replay, a signed bundle as the only input to the applier, and no human policy-write role in production.

Options weighed

• ChosenA registry compiled into engine policy: People review intent; machines write enforcement.
• RejectedHand-written engine policy with code review: Direct and flexible. Reviewers approve a policy language rather than a list of who gets what.
• RejectedA request-time policy engine in the gateway: Expressive and familiar where OPA is standard. Two policy engines deciding one request, and the environment boundary back inside policy.
• RejectedGrants through a web console: Friendly. No diff, no replay and no revert as one operation.

Consequences

What it buys

• Every grant has an owner, an approver and a commit
• A bad push is caught by replay before it applies, and reverted in minutes if not
• Nobody in production can change policy by hand

What it costs

• Grant changes take a pipeline run rather than a click
• The compiler and replay are platform code to maintain
• The applier identity is powerful and must be guarded

Choose differently when

An organisation already standardised on a request-time policy engine across its estate might keep that engine and compile the registry into it, keeping the environment boundary in the handshake.

Why it holds up over time

The registry outlives engines. Changing the engine or its policy language changes the compiler, while ten thousand approvals remain valid as written.

How does the platform know, for anything that could be compromised, exactly what that compromise would yield?

Context

The requirement makes the enumerated reachable set the definition of blast radius, per identity and per node, and asks for review of any identity above a threshold, because a workload that needs very many secrets is usually several workloads.

Decision

On every apply, the inventory computes each identity's reachable set from the compiled grants, each node's set from the pods scheduled on it, and each broker shard's set from its roles and admin credentials. Sets are published, shown in pull requests and used to preview revocation scopes. An identity reaching more than 40 secrets and roles is flagged for review. A separate report lists grants unused for 90 days, from the audit store, so reachable sets shrink over time.

How it works on-premise

The inventory service in Go on PostgreSQL, reading compiled policy and the estate's pod schedule every five minutes.

Options weighed

• ChosenComputed from compiled policy: Exact, current and derived from what is enforced.
• RejectedPeriodic manual access review: Familiar to auditors. Slow, sampled and out of date on the day it finishes.
• RejectedDerived from observed use: Shows what is needed. Used is not the same as reachable; kept as the unused-grant report.

Consequences

What it buys

• Every blast-radius claim in this record can be checked by a query
• Over-granted identities surface at review time, not during an incident
• Responders scope revocation with real numbers

What it costs

• Another service on the governance plane
• The node view is as fresh as its five-minute schedule read
• Threshold reviews create work for teams with legitimately broad needs

Choose differently when

None likely. The alternative is not knowing, and that is not a position the requirement allows.

Why it holds up over time

It is a join over identities, grants and placement. The computation survives any engine or orchestrator that can say who is granted what and what runs where.

When the audit sink is unavailable, which requests must be refused and which may continue?

Context

The requirement asks that every read be recorded before the material is returned; that custodial reads and human access fail closed when the audit sink is unavailable; and that high-volume brokered renewals fail open with a recorded gap. OpenBao's audit devices are all or nothing: if every device fails, every request fails.

Decision

For custodial reads, human access and policy changes, the gateway writes the record to Kafka with acks=all and returns material only after the acknowledgement; without it the request fails and the material is zeroed. For mints and renewals, OpenBao writes to a local audit log on NVMe before replying, which it will not skip, and Fluent Bit ships the log to Kafka; if shipping stalls or its disk buffer overflows, a gap marker with the missing range is written as soon as the stream is reachable. The KMS keeps its own audit devices for every unwrap.

How it works on-premise

Kafka in KRaft mode, three brokers per site, replication factor three and minimum in-sync replicas two. Local audit logs on dedicated NVMe with 72 hours of capacity. Fluent Bit with a filesystem buffer.

Options weighed

• ChosenPosture declared per event class: Refuses what must not go unrecorded, and keeps running what must not stop.
• RejectedEverything synchronous to the stream: Uniform. An audit outage becomes an estate outage through renewals.
• RejectedEverything asynchronous: Highly available. A custodial read could complete with no surviving record.
• RejectedOpenBao's local audit device only: Built in and blocking. Records sit on hosts administered by the platform team.

Consequences

What it buys

• No custodial or human read completes without a durable, off-host record
• An audit outage does not stop running workloads
• Gaps are records in themselves, not silent losses

What it costs

• Kafka is on the custodial read path
• Brokered events can reach the immutable store up to 30 seconds late
• Two audit paths to reason about and test

Choose differently when

If custodial read volume became very high and Kafka's latency a problem, the synchronous write could go to a local quorum log on the gateway hosts, shipped onward, with the same fail-closed rule.

Why it holds up over time

Declaring a failure posture per class of event is a requirement-level decision. It remains true whatever carries the events.

Where is the audit record kept so that it can be queried quickly, retained for years and altered by no platform operator?

Context

The requirement asks for 400 days queryable, seven years archived, human access records immutable for seven years, a single query answering who read a secret in a window, and a store the platform's operators cannot alter. At about 13 million brokered events a day per site, the renewal stream's retention is also the largest single cost.

Decision

Kafka feeds ClickHouse, ordered by secret path and time with a projection by actor, with a 400-day TTL. A chain sealer writes hourly hash-chained Parquet segments per partition to a Ceph RGW bucket with Object Lock in compliance mode for seven years. The records team administers ClickHouse and the Ceph cluster; the platform team holds producer rights only. Audit retention is reported as its own cost line.

How it works on-premise

ClickHouse ReplicatedMergeTree on three nodes per site with ClickHouse Keeper. A dedicated Ceph cluster for records at site B, replicated to site A. Chain anchors are also printed into the monthly security report so an edit would have to rewrite a document outside the system.

Options weighed

• ChosenClickHouse for query, Ceph Object Lock for retention: Fast structured queries, cheap compression and retention no platform role can shorten.
• RejectedOpenSearch: Good full-text search. Much more storage per event for this shape, and retention is easy to change.
• RejectedLoki: Cheap log storage. Weak at structured questions like every identity that read a path.
• RejectedPostgreSQL: Familiar. Billions of rows over 400 days is the wrong workload.
• Right elsewhereWORM tape for the archive: Cheapest cold retention. Right for organisations with a tape estate; retrieval takes days.

Consequences

What it buys

• The first question of every incident answers in seconds
• Evidence survives the platform and its operators
• Tampering with a segment breaks a chain whose anchors live elsewhere

What it costs

• Two storage systems owned by another team
• Seven years of compliance-mode retention cannot be shortened, even by mistake
• Renewal events dominate storage and must be watched

Choose differently when

An organisation with an existing audit log service that meets these retention and immutability rules should send events there and not build a second one.

Why it holds up over time

Hash-chained Parquet in S3-compatible object storage is readable by any analytics tool of the next decade. The record does not depend on ClickHouse, Kafka or this platform existing.

How does a human ever read a secret, and how is the platform's own staff treated as a threat?

Context

The requirement treats operators as an in-scope threat: no standing read access to material, break-glass with second authorisation, a stated reason, a time bound and an incident-grade record, and no operator credential able both to read material and to delete the record of reading it.

Decision

No human role has a policy on any data path. Platform SREs hold system operations only; security responders hold revocation and metadata. A human read goes through the access broker: sign-in with WebAuthn, a request naming the secret, the reason and the incident, a synchronous record, approval by a different person in the approver group, then a response-wrapped read valid once for 15 minutes. The unwrap pages security. Every break-glass read schedules an emergency rotation of that secret.

How it works on-premise

Keycloak federated with the corporate identity provider for operators, with hardware-key WebAuthn required. The access broker in Go on the governance cluster, with its own OpenBao identity allowed only to create wrapped reads. Non-exportable transit keys cannot be read by this path at all.

Options weighed

• ChosenAccess broker with wrapped single-use reads: Expensive, rare and loud, and self-alarming if intercepted.
• RejectedStanding read for a small security team: Fast in an emergency. A standing credential that reads everything is the target.
• RejectedEngine-native multi-person approval: No extra service. Ties the approval workflow to one engine's features and releases.
• RejectedShared break-glass account in a safe: Traditional. Anonymous, reusable and usually tested once.

Consequences

What it buys

• A compromised operator account causes an outage at worst, never a read
• An intercepted wrapping token announces itself
• Every human read ends in a rotation

What it costs

• Human access takes minutes and two people
• Emergency rotations follow every break-glass read
• The access broker is on the path of every incident that needs material

Choose differently when

None likely for production. In development partitions a single approver is acceptable and recorded.

Why it holds up over time

Two-person control, a stated reason and a single use are organisational rules that have outlived every access tool that has implemented them.

How does the platform notice a leaked credential that its legitimate holder is still renewing?

Context

Expiry contains a credential whose holder has died. A credential whose holder is alive is renewed, and a leaked copy stays valid for up to 12 hours. The requirement asks for detection of use inconsistent with the workload a credential was issued to, and for decoy credentials whose use is a confirmed incident.

Decision

Upstream authentication logs (PostgreSQL connection logs, RabbitMQ authentication events, partners' key-usage reports) are shipped to ClickHouse and joined on username to lease, session, pod and node. An alert fires when a credential is used from an address that is not its holder's pod, after its lease closed, or from outside the environment's networks. Decoy identities mint real accounts that can log in and have no privileges, and their credentials are placed where a thief would look. Any login with a decoy is a confirmed incident.

How it works on-premise

Log shippers on upstream hosts with the owners' agreement, a daily join of pod addresses from the estate's IP address management, and decoys refreshed monthly so they look current.

Options weighed

• ChosenJoin upstream use to leases, plus decoys: Detects exactly the case expiry cannot contain.
• RejectedNetwork intrusion detection alone: No upstream integration. Cannot tell a legitimate connection from a stolen one with the same credential.
• RejectedRely on expiry and revocation: No additional systems. Leaves live-holder leaks undetected for up to 12 hours.

Consequences

What it buys

• Live-holder leaks become detectable
• Decoys produce findings with no false positives
• The same join answers where was this credential used during an investigation

What it costs

• Integration with every upstream's logging
• Address attribution breaks down behind NAT and connection poolers
• Decoys must be maintained so they stay believable

Choose differently when

Upstreams that accept workload identity directly can check the presenter's identity themselves, which makes this join unnecessary for them.

Why it holds up over time

Matching use to issuance is independent of any product, and the join keys (usernames that encode leases, pod addresses) are under the platform's control.

How does a platform on every workload's start-up path change without becoming the outage?

Context

A defect in the gateway, the agent or an engine upgrade can stop every new pod in an environment. The invariants that matter most (no lifetime extension, no environment variables, no material in logs) are exactly the ones a well-intentioned change breaks without any functional test noticing.

Decision

Every release passes the invariant suite, a material scrubber over logs, traces and error messages, and a storm replay. It then soaks for 48 hours in staging at both sites, goes to one production partition at site A one node at a time, and reaches site B only after 24 hours clean. Agents roll out to 1%, 10% and then all nodes. No production change happens during a SEV1 or SEV2, or during a scheduled estate-wide redeploy.

How it works on-premise

Reproducible Go builds in Forgejo Actions, cosign signatures and SBOMs, Ansible for enclave hosts with signed artefacts staged on-host before the change window, and Argo CD for the agent DaemonSet. The previous binaries stay staged for a one-command rollback.

Options weighed

• ChosenPartition by partition with invariant gates: Each step risks one blast radius, and the properties that matter are tested explicitly.
• RejectedBlue-green by site: Fast to switch. Doubles capacity and exposes a whole site to a defect at once.
• RejectedTraffic-percentage canaries at the gateway: Good for stateless services. Engines are stateful clusters that cannot run two versions per request.
• RejectedContinuous deployment to production: Short lead time. Wrong for a Tier 0 dependency of the whole estate.

Consequences

What it buys

• A defect is found in one partition, not everywhere
• Invariants are guarded by tests rather than by memory
• Rollback needs no network fetch

What it costs

• A release takes several days to reach every partition
• Staging must be realistic enough for a soak to mean something
• Change freezes during incidents can delay fixes

Choose differently when

Security fixes for actively exploited vulnerabilities follow an expedited path with the same order and shorter soaks, approved by the security lead.

Why it holds up over time

Staged rollout and property tests are practices, not products. They carry across any change of language, engine or deployment tool.

How are the costs of security choices made visible without punishing teams for choosing safely?

Context

Lease lifetime is the platform's main load parameter. Shorter leases mean more renewals, more account churn upstream and more audit rows, and the renewal stream's retention is often the largest cost in the system. The requirement asks for attribution per team, environment and secret, the cost implied by each lifetime, upstream cost as well as platform cost, audit retention as its own line, and cache hit rate reported with cached reads free.

Decision

The lifetime model publishes renewals, mints, account churn, audit rows and storage for every role's chosen values, and the registry pull request shows the change. Monthly reports attribute requests, mints and audit volume per team, environment and secret, including upstream accounts and connections consumed. Costs are shown, not charged. Cache hit rate sits on the platform's headline dashboard, and reads served from cache cost nothing.

How it works on-premise

Attribution is computed in ClickHouse from the audit stream joined to the registry's ownership data. Unit costs come from the enclave's hardware and power share, and upstream costs are supplied by the upstreams' owners.

Options weighed

• ChosenShowback per team with lifetime modelling: Owners see the price of a shorter lease at the moment they choose it.
• RejectedCharge per request: Precise. It rewards longer leases, which is paying teams to be less safe.
• RejectedA flat platform charge: Simple. Hides the renewal and retention costs that owners can actually influence.

Consequences

What it buys

• Security and cost trades are made with numbers in front of the person making them
• Retention of renewal events is visible as the cost it is
• Clients bypassing the agent show up as falling cache hit rates

What it costs

• Attribution pipelines to maintain
• Showback without chargeback relies on steering-group attention
• Unit costs for on-premises hardware are estimates and invite argument

Choose differently when

An organisation with mature internal chargeback may charge per team, provided it charges for audit retention and upstream consumption rather than for request count.

Why it holds up over time

The model is arithmetic over declared lifetimes and observed volumes. It stays correct as prices, hardware and engines change, and is regenerated from telemetry rather than rewritten.