Architecture One-Pager

Secrets Distribution Service · Solution Architecture v1.0 · Security Platform Architecture · 2026-09 · 34 views · open source, fully on-premises

No workload holds a secret it could leak for long, and no credential anywhere depends on someone remembering to revoke it.

Every estate has the same secrets problem in different clothes: a database password in a CI variable, a partner key in a Helm values file, a signing key on a build agent's disk. Each one was put there because a workload needed it at start-up and there was nowhere better to put it. A vault on its own does not fix this. It moves the password into the vault and puts the vault's token where the password used to be. That token is secret zero, and a platform that has not said where secret zero comes from has centralised the problem, not removed it, and made the remaining secret more valuable. The second failure is quieter. A credential that never expires relies on revocation to contain a leak, and revocation relies on someone noticing the leak, finding every copy, and an upstream that honours the call. All three fail regularly and at the worst time. This platform exists to make two things true together: a workload authenticates with what the infrastructure can attest about it, and whatever it receives expires without anyone's help.

A pod starts with nothing. On its node, a SPIRE agent asks the kubelet what the pod is (namespace, service account, image digest) and a built credential agent obtains that pod's SPIFFE identity through SPIRE's Delegated Identity API. The agent speaks to an issuance gateway as the pod, over mTLS, presenting a five-minute JWT-SVID bound to the same key. The gateway is small, stateless and built in Go. It accepts only certificates from the environment's own trust domain, wraps the engine's token into a session bound to the pod's key and to the partition's restore generation, prioritises renewals over first issuance, and decides how each event class is audited. Behind it, every environment at every site has its own OpenBao clusters: broker shards that mint per-instance accounts on PostgreSQL, MariaDB, MongoDB, RabbitMQ, Kafka and Ceph and hold the leases that revoke them; a custody cluster that holds partner keys and other unmintable material as ciphertext; and a KMS cluster on an HSM seal, run by key custodians, that unwraps custodial material on every read and holds the restore generation. Credentials reach applications through a local socket by default, a tmpfs file for software that cannot change, or an Envoy proxy that attaches an API key the application never sees. Every attestation, mint, read and revocation is recorded in Kafka before or as the credential is returned, kept for 400 days in ClickHouse and seven years in Ceph Object Lock owned by the records team. No one writes policy: teams declare identities, secrets, roles and grants in a reviewed Git registry, and a compiler produces the SPIRE entries and OpenBao policies.

What it is, and what it is not

• Identity attested by the infrastructure at start-up — not a bootstrap token baked into an image or injected by the pipeline.
• Credentials minted per workload instance that expire on their own — not a shared service-account password rotated by a quarterly change ticket.
• Custody and brokerage offered as two products with two guarantees — not custodial secrets described as if they expired like minted ones.
• Environment separation decided in the TLS handshake — not environment separation written as a policy that one bad merge can undo.
• Fail static until expiry, with the price stated — not an emergency grace period that quietly makes every stated lifetime false.
• Revocation reported as contained when the upstream confirms it — not a green tick for an API call that returned success.

The decisions that are the architecture

1. The chain of trust ends at attestation (ADR-01) — Nodes prove themselves with a token the Kubernetes control plane signs; pods are identified by what the kubelet observes, including the image digest. Every place the chain still rests on a bearer secret is written in a register with an owner.
2. Environments are separate trust domains (ADR-04) — A staging identity cannot complete a handshake with a production gateway. No policy anyone writes can grant it production access, because policy is never reached.
3. One memory image yields one partition (ADR-05) — KMS, custody and broker are separate clusters per environment and per site. A broker node's memory holds its upstreams' admin credentials and nothing custodial; a custody node holds ciphertext and no key.
4. Custodial material is unwrapped on every read (ADR-11) — By the gateway's identity, at a KMS the platform team does not run. The KMS log is a complete record of custodial use, and a KMS outage fails custodial reads closed by construction.
5. A thin gateway enforces what the engine cannot (ADR-09) — Key-bound sessions, restore generations, environment checks, renewal priority and per-class audit posture live in one small stateless service that can be deleted piece by piece as engines catch up.
6. Silence revokes (ADR-15) — Leases hang off a session that the agent renews only while the pod exists. A pod that vanishes without a word loses every credential within 15 minutes, and the upstream's own VALID UNTIL is a third clock.
7. Fail static until expiry, and no switch to extend (ADR-21) — During an outage, agents serve what they hold until it expires and no longer. Availability is bought with the control plane's own availability, not with lifetimes.
8. Nobody writes policy (ADR-27) — People declare intent in a reviewed registry; a compiler writes SPIRE entries and OpenBao policies; every change is replayed against a week of real traffic before it applies.

Why it holds up over time

A secrets platform outlives several generations of engines, HSMs, orchestrators and cryptographic algorithms, and it will be operated by people who were not in the room for any of these decisions. The design keeps its promises in standards, in data and in tests, and keeps every product behind a seam narrow enough to replace without asking a single application team to change code.

• The identity contract is a standard, not a product. Workloads are named by SPIFFE IDs and prove it with SVIDs. SPIRE can be replaced, and node attestation can move from service-account tokens to TPMs, without renaming a single identity or re-approving a single grant.
• Applications depend on the agent, not the engine. The socket protocol is the only interface any team codes against. OpenBao could be exchanged for a successor fork, a relicensed Vault or a built engine behind the gateway, and no workload would notice.
• The brokered tier is designed to shrink. Where an upstream learns to accept workload identity directly, through mTLS, OIDC federation or STS web identity, its roles are retired. The platform measures success by credentials that no longer need to exist, which keeps it from becoming a moat.
• Policy is compiled from intent. Grants are reviewed as plain declarations of who needs what. A change of engine or policy language changes the compiler, not ten thousand approvals.
• Cryptography can move without re-issuing secrets. Envelope keys are versioned and ciphertext is rewrapped in place. A new HSM, a key-size change or a post-quantum algorithm is a rewrap job that never exposes plaintext to storage.
• Separation of duty is organisational. The KMS and the audit archive are run by different teams from the platform. That survives re-platforming far better than any technical control, because it does not depend on how the technology is wired.
• The invariants are tests. No lifetime extension, no credential in an environment variable, no material in a log. They run on every build, so they survive team turnover and the well-meaning refactor that would otherwise erode them.
• Every remaining static credential has a name. The exception register turns the unsolved part of the problem into a backlog with owners. Estates keep inventing new ways to hide a password; a register makes each one visible instead of letting it become folklore.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.

Scope

In scope

• Workload attestation, registered identities, key-bound sessions and the secret-zero exception register
• Brokered credentials for databases, message brokers, object storage and internal token endpoints, with leases, renewal and revocation
• Custodial storage, versioning, rotation with overlap, maximum age and non-exportable cryptographic operations
• The node agent and its three delivery modes, including support for software that cannot be changed
• Revocation by lease, identity, node, role, namespace and partition, and orphan reconciliation
• Policy compiled from a reviewed registry, computed blast radius and break-glass human access
• The audit record, its retention and immutability, leak detection and decoy credentials

Explicitly out of scope

• The upstreams themselves and their account models; the platform integrates and never operates them
• TLS certificates for service-to-service traffic, which belong to the service mesh
• Human sign-in to applications and the corporate identity provider
• Code-signing policy and supply-chain verification, beyond holding signing keys as non-exportable
• Any managed cloud service; everything runs on hardware the organisation owns

Prove it before production depends on it

Eight weeks, two sites, one real PostgreSQL fleet and one RabbitMQ cluster on staging, and three volunteer teams. Each item replaces an assumption in this record with a measurement.

1. Seal the KMS cluster with the chosen HSM pair on the chosen OpenBao release, pull one HSM, restart all nodes, and time recovery
2. Drive one broker shard at 2,000 lease writes a second for four hours and measure commit latency; kill the leader with 90,000 leases outstanding and time until revocations resume
3. Run 110 pods on one node through the Delegated Identity API with SVID rotation and time every attestation
4. Mint, renew and revoke 10,000 PostgreSQL roles with open sessions under the scoped admin role and record the p99 of confirmed revocation
5. Rotate a credential under Envoy's credential injector during a load test and count failed requests
6. Replay a full node-pool replacement against staging and record the time to steady state, split by upstream mint budget
7. Restore a broker snapshot 40 minutes old and verify that no revoked account returns and no pre-restore session is honoured

• Pass: four weeks of synthetic attest, mint, read and revoke from both sites inside every latency target, with zero material found by the scrubber.
• Pass: a site partition taken down for an hour degrades exactly as view 23 states, and no credential outlives its stated expiry.
• Fail and rethink: one broker shard cannot sustain 1,000 lease writes a second at the target latency, which moves lease state to a horizontally scalable store with OpenBao retained for custody and transit.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 34 records across 9 areas, each with the alternatives that lost and what the choice costs.