# Secrets Distribution Service

**Solution Architecture v1.0 · open source, fully on-premises · Security Platform Architecture · 2026-09 · 34 views · 34 architecture decision records**

The service a workload calls before its first query, and again before its credential lapses. It proves what the workload is, decides what it may have, mints a credential or releases a custodial secret, delivers it to the process, and makes sure it expires without anyone's help, with revocation by scope as the fast path. It serves 40,000 workload instances across three environments and two data centres the organisation owns. It is built from open-source software: SPIRE for attested workload identity; OpenBao split into separate KMS, custody and broker clusters per environment and site, with the KMS on an HSM seal run by key custodians; a stateless issuance gateway and a per-node credential agent built in Go; Kafka, ClickHouse and Ceph Object Lock for the audit record; and a Forgejo registry from which every policy is compiled.

The design rests on one rule: a workload proves what it is with something it neither stores nor was given, and receives something that expires on its own. Neither half is optional.

The decisions that carry the design:

- **The chain of trust ends at attestation.** Nodes attest with tokens the Kubernetes control plane signs; pods are identified by namespace, service account and image digest. Every place that still rests on a bearer secret is registered with an owner.
- **Environments are separate trust domains.** A staging identity cannot complete a TLS handshake with a production gateway, so no policy can grant it production access.
- **One memory image yields one partition.** KMS, custody and broker are separate clusters per environment and per site.
- **Custodial material is unwrapped on every read**, by the gateway's identity, at a KMS the platform team does not run. A KMS outage fails custodial reads closed by construction.
- **A thin gateway enforces what the engine cannot:** key-bound sessions, restore generations, renewal priority and audit posture per event class.
- **Silence revokes.** Leases hang off sessions that the agent renews only while the pod exists.
- **Fail static until expiry, with no switch to extend.** The price, a site's credentials lapsing after 60 minutes of outage, is stated rather than engineered away.
- **Nobody writes policy.** Teams declare intent in a reviewed registry; a compiler writes SPIRE entries and OpenBao policies and replays a week of real traffic against each change.

The architecture one-pager (including why the design holds up over time) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~12 min) and [docs/decision-record.md](docs/decision-record.md) (~90 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 34 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 34 decision records, the technology table and the package glossary as markdown |
| `specs/part-a..f.json` | Diagram specifications, the source of truth for every view, with every icon pinned |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## Rebuild

```bash
bash scripts/build.sh
```

The build needs Node 20+ and the `architecture-diagram-web` skill. Every view passes the geometry check (0 errors, 0 warnings) and the routing check (0 errors, 0 warnings). Every icon resolves (0 unresolved, 0 weak), and every relative link and in-page anchor resolves.

## Notes on the requirement

- **Stack.** The requirement targets Google Cloud. As requested, this package runs on premises with open source. GKE Workload Identity Federation and the STS exchange become SPIRE attestation with Kubernetes projected tokens, exchanged at the gateway for a key-bound session (ADR-01, ADR-03). Cloud KMS becomes an OpenBao KMS cluster on an HSM seal, run by key custodians (ADR-10). Secret Manager becomes an OpenBao custody cluster holding envelope-encrypted ciphertext (ADR-11). The self-hosted broker on GKE becomes OpenBao broker shards on dedicated enclave hosts rather than Kubernetes, because nearly everything in the estate needs the platform in order to start (ADR-06). Spanner becomes OpenBao's integrated Raft per site, since lease state is regional by requirement (ADR-07). Cloud Audit Logs into BigQuery becomes Kafka into ClickHouse and Ceph Object Lock, owned by the records team (ADR-29, ADR-30).
- **The throughput figures do not reconcile with the lease figures, and the design says so.** 180,000 leases renewed at two-thirds of a 30-minute lifetime produce about 150 renewals a second per site, and the whole steady state is roughly 550 requests a second. The stated 6,000 rps at 85% renewals implies renewal every 35 seconds, which is what clients without an agent look like. The gateway is sized to the stated figures anyway, and the proof phase replaces both with measurements (view 28). The stated burst is about thirteen times a full estate restart; in practice a storm is bounded by how fast upstreams create accounts, not by the platform (view 22).
- **Short leases do not contain a leak while the holder lives.** A renewed credential stays valid for its total lifetime, 12 hours by default, however short each lease is. Expiry contains leaks from workloads that have died; revocation by scope and Phase 3 leak detection cover live ones. High-value roles can set a shorter total lifetime at a visible churn cost (ADR-14, ADR-32).
- **Key binding and per-class audit posture are built, not configured.** OpenBao tokens are bearer tokens, and its audit devices fail every request or none. The gateway supplies key binding, the restore generation and the synchronous audit write for custodial and human reads, while OpenBao's local audit log carries the brokered path (ADR-03, ADR-09, ADR-29).
- **Multi-site moved into the MVP.** The requirement puts multi-region in Phase 2. A dependency on every workload's start-up path cannot serve production from one site, so the MVP may serve staging from one site and production only from two (view 29).
- **Custodial continuity.** The requirement bounds agent continuity at 60 minutes but custodial secrets have no lease. The agent evicts a custodial copy 60 minutes after reading it, so the continuity bound is the same for both shapes (ADR-21).
- **Rotation's last mile.** Retirement waits on observed use with a seven-day ceiling, after which a named straggler is broken deliberately. The requirement's 24-hour window is treated as a floor, not a ceiling (ADR-23).
- **Unproven behaviours.** Seven product behaviours must be confirmed on the chosen releases and hardware before production depends on them, including OpenBao's HSM seal, the single-writer lease ceiling and SPIRE's Delegated Identity API at 110 pods per node. They are listed in the decision record's status note, and each has a proof-phase item in the one-pager.