# Architecture One-Pager

*API Key and Token Service · Solution Architecture v1.0 · Google Cloud with open-source edge components · Security Platform Architecture · 2026-09*

**The credential carries no authority of its own. It is an opaque pointer into revocable state, and verification is a local read of a replicated, staleness-bounded projection of that state.**

A developer platform hands its customers a string and then has to live with it. The string ends up in a CI variable, a mobile binary, a pasted Slack message and, eventually, a public commit. On the day it does, the only question that matters is how long it stays useful to whoever found it — and the answer is decided years earlier, by whether the platform chose a credential it can take back. Meanwhile every one of the platform's 250,000 requests per second has to check that string, which means the check has to be cheap enough that nobody is ever tempted to skip it under load.

Issuance is a rare, transactional write into a strongly consistent, tenant-partitioned store, and it returns the secret exactly once. What is stored is a keyed digest whose pepper lives in a hardware key store the database cannot reach. Revocation is appended to a separate, ordered, append-only log with RPO 0 and acknowledged only after a durable multi-region write. That log is published as a feed and applied as an additive overlay onto a projection held inside every calling service, beside a snapshot that warms a cold one. Verification is then a memory lookup and an overlay check on localhost: no network hop, no call to the control plane, and a staleness figure returned on every response. Past a declared staleness ceiling a verification point stops honouring privileged scopes rather than continuing to answer from a projection it can no longer prove is fresh.

## What it is, and what it is not

- **An opaque pointer into revocable state** — not a signed token carrying its own scopes, valid until it expires whatever anyone does.
- **Verification as a local memory read** — not a network call to a central authority that callers will cache badly or skip under load.
- **Revocation acknowledged only after a durable multi-region write** — not acknowledged on receipt and hopefully replicated before the next failover.
- **Staleness measured, published and acted on** — not a cache with a TTL and no idea how far behind the truth it is.
- **A digest the database alone cannot attack** — not a hash in the same store as everything else, one dump away from an offline crack.
- **Scope decided at issuance, as the blast radius** — not a wide key issued quickly and narrowed later, which never happens.
- **Automated leak response that is reversible and attributed** — not an anomaly detector that takes a customer's production down and cannot say why.
- **Propagation time measured continuously in production** — not a propagation target written in a design document and never observed again.

## The decisions that are the architecture

1. **The credential is a pointer, not a claim** (ADR-01) — Opaque keys with a public prefix and identifier; nothing about what the key may do travels inside it. Revocation is possible at all only because of this.
2. **Verification is a local read** (ADR-02) — A library or sidecar holds a projection of active credentials and a revocation overlay. The request path makes no call to the control plane, ever.
3. **Revocation is its own durable, ordered log** (ADR-05) — Separate from credential replication, RPO 0, acknowledged only after a multi-region quorum write, and never undone by a restore or a cache repopulation.
4. **Staleness is explicit and bounded** (ADR-07) — Every response carries how far behind the feed it is; past 120 seconds a point refuses write and admin scopes rather than answering from state it cannot vouch for.
5. **A fast keyed digest, peppered outside the store** (ADR-09) — HMAC-SHA-256 with a Cloud HSM pepper, not Argon2id. 160 bits of platform entropy is not a password, and 250,000 verifications per second cannot afford a memory-hard KDF.
6. **Compromise of the store is a pepper rotation** (ADR-10) — Versioned pepper with dual-version verification, so a full database copy is answered by rotating one key rather than re-issuing every customer's credentials.
7. **Scope is the blast radius, fixed at issuance** (ADR-13) — A flat, additive vocabulary; no principal may issue a credential wider than itself; a scope change is a new version, not an edit.
8. **last_used_at is an observation, never an input** (ADR-15) — Lossy, lagging and cheap, because promoting it to an authorization input would make the hot path a durable write at full request rate.
9. **Automated containment is reversible and attributed** (ADR-17) — Auto-revoke on a confirmed partner match; anomaly-only response is opt-in, because a false positive is an outage the customer did not cause.
10. **Propagation is measured, not asserted** (ADR-19) — A canary issues, verifies, revokes and re-verifies a credential continuously, from a project this platform's operators cannot deploy to.

## Why this should still be right in ten years

A credential service outlives the platform it was built for, because every customer integration depends on the format of the string. These are the properties that should survive a change of scale, of cloud, and of the people who built it.

- **The pointer-versus-claim boundary is technology-independent.** Nothing in ADR-01 depends on Spanner, on Google Cloud or on HMAC. The claim — that authority is looked up rather than carried — survives replacing every component underneath it, and it is the property a future team would most easily throw away without noticing.
- **Propagation time is the metric that ages well.** Scale changes what the architecture looks like; it does not change what the service is judged on. A team in ten years with a different store, a different bus and ten times the traffic will still be asked how long a revoked key stays useful, and the canary will still answer.
- **Staleness is a first-class output, not an implementation detail.** Because every response carries it, any future replacement of the propagation mechanism — filter, push, pull, something not yet invented — is testable against the same contract. The interface outlives the mechanism.
- **The secret's unrecoverability is structural.** It is not enforced by a policy that can be relaxed or an ACL that can be misconfigured. A future operator cannot accidentally make secrets recoverable without changing the issuance path itself, which is a visible, reviewable act.
- **The scope vocabulary is additive by construction.** Removals are rejected at the gate rather than deprecated by convention, which means a vocabulary ten years and four owners later is still one a customer's five-year-old integration can rely on.
- **The decisions most likely to be wrong are the ones with numbers on them.** The staleness ceiling, the overlap window and the projection-fits-in-memory assumption are each named, each measurable, and each carry a stated condition that would flip them. They are designed to be overturned by data rather than defended.

## Non-functional targets

Every number here is a stated assumption from the requirement, chosen to be argued with. The right-hand column names the view where the mechanism that meets it is drawn.

| Quality | Target | How it is met | View |
|---|---|---|---|
| Verification availability | ≥ 99.99% monthly, per region | Active-active verification in every region, with no per-request dependency on the control plane or the credential store. | 16 |
| Verification latency | p50 ≤ 0.3 ms · p99 ≤ 3 ms in-process | Memory lookup of a keyed digest plus an overlay check on localhost; a keyed HMAC rather than a memory-hard KDF. | 12 |
| Verification throughput | 250,000/s · 5× burst for 120 s | Verification scales with the calling fleet; no coordination between verification points. | 08 |
| Control-plane availability | ≥ 99.9% monthly | Leader region with warm standbys; issuance failing is survivable in a way verification failing is not. | 16 |
| Revocation propagation | p50 ≤ 2 s · p99 ≤ 10 s · ceiling 60 s | Ordered append-only log published as a push feed with a 5-second poll fallback and a heartbeat. | 14 |
| Bulk revocation | 100,000 credentials within 5 min | Tenant epoch bump: one ordered write and one message, whatever the number of credentials. | 14 |
| Leak response | p99 ≤ 60 s from partner notification | Greppable prefixes, a one-bit partner endpoint, and a per-tenant response policy executed without a human in the loop. | 15 |
| RPO — revocation state | 0 | Multi-region quorum write before the revocation is acknowledged; never undone by replication catch-up or restore. | 10 |
| RPO — credential records | ≤ 5 s | Multi-region Spanner configuration; a lost issuance is recoverable by re-issuing, a lost revocation is not. | 10 |
| RTO — verification path | 0 | Regional loss is a capacity event; every region verifies independently from its own projection. | 16 |
| RTO — control plane | ≤ 15 min | Warm standby regions; failover promotes a standby without changing the verification path. | 16 |
| Audit retention | 400 days hot · 7 years cold | Hash-chained records in BigQuery, tiered to locked Cloud Storage; the trail outlives the credential. | 10 |
| Time to first API call | ≤ 2 min at p50 | Self-service issuance with scope presets, one-time secret display and environment encoded in the prefix. | 04 |

## Scope

**In scope**

- Issuance of opaque API keys, restricted keys and short-lived tokens minted from them.
- The scope vocabulary, its versioning, and the no-escalation rule at issuance.
- Verification on every request, as a library, a sidecar and a remote fallback.
- Revocation, rotation with overlap, expiry and the propagation that makes them real.
- Leak detection integration, automated containment policy, and the decision trail behind it.
- Lifecycle audit, usage attribution per credential, and export to a tenant's own SIEM.

**Explicitly out of scope**

- End-user authentication, sessions, consent screens and SSO.
- What a scope actually permits in any product surface — that is product authorization, and it reads this service's output.
- Rating and billing. Usage is attributed to a credential and handed over.
- Customer-side secret storage. The platform can tell a customer their key leaked; it cannot manage where they put it.
- Certificate issuance and mutual-TLS identity for workloads, which is a different problem with a different lifecycle.

## What a four-week prototype should prove

Three of this architecture's claims are cheap to test and expensive to be wrong about. A prototype that proves these can be built on and one that skips them is a demonstration of issuance, which is the easy half.

1. End-to-end propagation: issue, verify, revoke, re-verify, with the p99 measured across three regions and a deliberately partitioned fourth verification point.
2. Projection memory at scale: 2.5 million synthetic credentials loaded into one verification point, measured, and the number compared against the smallest pod the platform runs.
3. Digest cost under load: HMAC verification at 250,000/s on the target instance type, with the p99 measured while the overlay is being updated.
4. Cold-start behaviour: a verification point restarted with no projection, with admission gated on readiness, measured against the credential store's load.

- Kill the revocation feed and confirm that every verification point crosses the staleness ceiling and refuses privileged scopes, rather than continuing silently.
- Restore the credential store from a backup taken before a revocation and confirm the revocation survives.
- Rotate the pepper and confirm dual-version verification holds without re-issuing a single credential.

## Open risks, carried rather than hidden

| Risk | If it lands | Response |
|---|---|---|
| Full projection does not fit | The memory-per-point assumption fails at a few thousand verification points, forcing a demand-filled cache and reintroducing a cold path to the store. | Measure in the prototype. The fallback is a partitioned projection by key-prefix shard, which keeps the no-cold-path property at the cost of routing. |
| Fail-closed is politically unsurvivable | The first time a feed outage refuses writes platform-wide, the pressure to fail open will be enormous, and failing open silently undoes the architecture. | Scope-graded degradation is the compromise already designed in. Any move beyond it should require the same review as removing revocation entirely. |
| Central scope vocabulary becomes a bottleneck | Product teams route around it with catch-all scopes, and least privilege quietly stops being true. | Watch the rate of new scopes and the share of credentials holding an admin-grade scope. A rise in the second is the signal, not a complaint from a team. |
| Partner leak detection is the only detection | Keys leaked to private repositories, logs and support tickets are invisible, and the platform's containment metric looks better than its containment. | Own-corpus scanning is in the must-have tier for exactly this reason; report detection source split rather than a single leak count. |
| Anomaly detection is turned on by default under pressure | A false positive revokes a live customer integration, and the trust cost exceeds the leak it prevented. | Opt-in is an architectural decision, recorded in ADR-17 with its reasoning, so that turning it on by default is a visible reversal rather than a configuration change. |

The reasoning behind every component and technology choice is in the [Architecture Decision Record](decision-record): 20 records across 6 areas, each with the alternatives that lost and what the choice costs.