# API Key and Token Service

**Solution Architecture v1.0 · Google Cloud with open-source edge components · Security Platform Architecture · 2026-09 · 22 views · 20 architecture decision records**

The credential control plane and verification data plane for a developer platform: the service that issues the string a customer pastes into their CI config, decides on every one of 250,000 requests per second whether that string is still valid and what it carries, and takes it away faster than whoever found it in a public commit can use it. It holds 2.5 million live credentials across 400,000 tenants in three regions, and it is built from Cloud Spanner for credential and revocation state, Pub/Sub for propagation, Cloud KMS/HSM for the digest pepper, Cloud Storage for projection snapshots, Memorystore and Bigtable for the derived tiers, BigQuery for audit, and an Envoy `ext_authz` sidecar plus in-process libraries for verification itself.

The design rests on one rule: **the credential carries no authority of its own.** It is an opaque pointer into revocable state, and verification is a local read of a replicated, staleness-bounded projection of that state — never a signature check, and never a call to the control plane.

The decisions that carry the design:

- **Opaque, not self-contained.** Nothing about what a key may do travels inside it, which is what makes revocation possible at all. A signed token is the right answer for ninety seconds inside a trust boundary and the wrong one for a string that will appear in a public commit (ADR-01).
- **Verification is a memory lookup.** A sidecar or library holds a projection of active credentials and a revocation overlay; the request path makes no network call. A check that is expensive gets skipped, and nobody tells you (ADR-02).
- **Revocation has its own durability.** An independent append-only log with RPO 0, acknowledged only after a multi-region quorum write, and never undone by a restore. Losing an issuance is recoverable; losing a revocation is not (ADR-05).
- **Staleness is published and bounded.** Every response says how far behind the feed it is. Past 120 seconds a verification point refuses write and admin scopes rather than answering from state it cannot vouch for (ADR-07).
- **Bulk revocation is an epoch bump.** One ordered write and one message, whether it kills one credential or a hundred thousand — so propagation is fastest exactly when the incident is largest (ADR-08).
- **A fast digest, peppered outside the store.** HMAC-SHA-256 under a Cloud HSM pepper, not Argon2id: 160 bits of platform entropy is not a password, and a memory-hard KDF at 250,000/s would make local verification impossible (ADR-09).
- **Store compromise is a pepper rotation.** The pepper is versioned with dual-version verification, so the answer to an exfiltration is rotating one key rather than asking every customer to re-issue (ADR-10).
- **The secret is unrecoverable by construction.** Not by policy, not by an access-control list. A restore of every store the platform operates cannot revive one secret (ADR-11).
- **Automated containment is proportional to evidence.** Auto-revoke on a confirmed partner match; anomaly-only response is opt-in, because a false positive is an outage the customer did not cause (ADR-17).
- **Propagation is measured, not asserted.** A canary issues, verifies, revokes and re-verifies a credential continuously, from a project this platform's operators cannot deploy to (ADR-19).

The architecture one-pager (including why the design should still hold up in ten years) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~10 min) and [docs/decision-record.md](docs/decision-record.md) (~67 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 22 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 20 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..f.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## The views

| # | View | Act | What it proves |
|---|---|---|---|
| 01 | System Context | Scope | Who uses it, who calls it on every request, and that product authorization sits outside the boundary |
| 02 | High-Level Architecture | Scope | Six stages from a developer pressing "create" to a leaked key being useless |
| 03 | Actors and Journeys | People | Six actors with goals in their own voice, and what each gets to do |
| 04 | Journey — the first API call | People | Signup to a 200 response, and why the one-time secret display stays |
| 05 | Journey — a key in a public commit | People | That the hard moment is Decide, not Detect — which makes blast radius structural |
| 06 | Layered Architecture | Structure | Six layers, and that no arrow in the data plane points upward |
| 07 | Control Plane Components | Structure | The services that issue, scope, rotate and revoke, and the state they own |
| 08 | Verification Data Plane | Structure | What sits in a calling pod, and that the credential store is not reachable from it |
| 09 | Integration Surface | Structure | Four inbound interfaces, three outbound, and the one that is unauthenticated on purpose |
| 10 | Storage Zones | Data | Authoritative, evidential, derived — and the box holding what is never stored |
| 11 | Data Model | Data | Nine entities, and the absence of a secret column in the middle of them |
| 12 | Critical Flow — verify a request | Runtime | Thirteen messages, none of which leaves the pod |
| 13 | Critical Flow — issue, then rotate | Runtime | The one moment the secret exists, and the overlap that makes losing it survivable |
| 14 | Revocation Propagation | Runtime | Accept, order, publish, apply, prove — and what happens when it cannot be proved |
| 15 | Leak to Containment | Runtime | Five lanes across six stages, and that in stage one nothing is wrong |
| 16 | Deployment Architecture | Operations | Active-active verification, single-leader control plane, two recovery objectives |
| 17 | Release and Config Pipeline | Operations | How a vocabulary change reaches the fleet without being able to break it |
| 18 | Observability | Operations | Golden signals, correctness, proof, silence detection, and where each is watched from |
| 19 | Credential Lifecycle | Operations | The loop every key travels, and where a leak cuts into it |
| 20 | Security Trust Zones | Assurance | Six zones, and the two crossings that deliberately do not exist |
| 21 | Identity and Access | Assurance | Who proves what before a credential exists at all |
| 22 | Failure Modes | Assurance | Eight failure classes, each with the price of the chosen response |

## Rebuild

```bash
bash scripts/build.sh
```

The build needs Node 20+ and the `architecture-diagram-portal` skill checked into this repository (override with `SK=<path>`). Every view passes the geometry check (0 errors, 0 warnings) and the routing check (174 edges, 0 errors, 0 warnings). Every icon resolves (325 nodes, 0 unresolved, 0 weak), 349 icons are embedded across the set, and every relative link resolves.

## Notes on the requirement

- **Stack.** The requirement names Google Cloud, and the rotation is the reason: of the twenty-four packages in this practice that name a stack, ten are open-source on-premises, eight are Microsoft Azure, two are open-source without a named venue, one is Amazon Web Services and one is a company's own estate. None had been delivered on Google Cloud — the one earlier requirement that named it, `storage-tiering-service`, was retargeted to an on-premises stack before the diagrams were drawn. A credential service belongs to no particular cloud, so the topic was free to rotate.
- **Every number is a stated assumption.** 400,000 tenants, 2.5 million live credentials, 250,000 verifications per second, the latency and propagation percentiles, the retention tiers and the recovery objectives are invented to be defensible and arguable for a mid-sized developer platform, not measured. Three are explicitly the first that real data should overturn: the 120-second staleness ceiling, the 7-day rotation overlap, and the assumption that a full projection fits in every verification point's memory.
- **The full-projection assumption is the package's largest single bet.** At 2.5 million credentials and roughly 160 bytes per entry it is about 400 MB per verification point, which fits the platform's standard pod and not its smallest. If the measurement fails, ADR-03's deferred option — a partitioned projection by key-prefix shard — becomes the design, and the prototype in the one-pager is built to falsify this first.
- **Fail-closed is the decision most likely to be reversed under pressure.** The first feed outage that refuses writes platform-wide will produce enormous pressure to fail open, and failing open silently undoes the architecture. The scope-graded compromise in ADR-07 is already the concession; the package says in three places that moving beyond it should require the same review as removing revocation.
- **The revocation canary is a control, not infrastructure.** It runs in a separate project, does almost nothing on a good day, and is the most likely line item to be cut in a cost review. Cutting it leaves the architecture's central claim — that revocation reaches every projection — unverified, and ADR-19 says so.
- **The prefix is the most permanent artefact here.** It appears in customer configuration everywhere and cannot be changed without breaking every scanning partner. That permanence is what makes the one-bit constraint on the partner endpoint (ADR-16) load-bearing rather than fastidious.
- **Two decisions are left genuinely open and are marked as such.** The revocation distribution mechanism (push, poll or probabilistic filter) has a named trigger — roughly ten thousand verification points — and scope-vocabulary ownership is a live organisational question the architecture cannot settle on its own. Both appear in the requirement's Core Architecture Questions and in the cards on views 14 and 17.
- **Act 2 is drawn from the requirement rather than decorated onto it.** The two journey maps have their troughs in different places, and the second one's trough is the finding: a security engineer's hardest moment is not detecting the leak but deciding to revoke without knowing what breaks, which is why usage attribution is a structural requirement rather than a reporting feature.