# Architecture Decision Record

*API Key and Token Service · Solution Architecture v1.0 · Google Cloud with open-source edge components · Security Platform Architecture · 2026-09*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Twenty decisions make up this architecture. Everything else across the twenty-two views is convention, sizing or consequence. Each record states the forcing question, the context that makes it hard, what was decided, how it is realised on Google Cloud, the options weighed, what the choice buys and costs, the conditions that would flip it, why it should still be right in ten years, and the lesson that transfers beyond this platform.

> **Status of this document.** This is a design, not a report on a running system. The rates, latencies, volumes, retentions and windows are the requirement's stated assumptions for a mid-sized developer platform — 400,000 tenants, 2.5 million live credentials, 250,000 verifications per second across three regions — invented to be defensible and arguable rather than absent. They are to be replaced by measured telemetry before build, and three of them in particular are the first numbers real data should overturn: the 120-second staleness ceiling, the 7-day rotation overlap, and the assumption that a full credential projection fits in every verification point's memory.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it is realised on Google Cloud:** The concrete mechanism: which service or package, configured how, in which project.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**Credential form and verification**: What a credential is, and what it costs to check one.

- ADR-01 · The credential is an opaque pointer, not a self-contained claim
- ADR-02 · Verification is a local read, shipped as a library and a sidecar
- ADR-03 · A full projection at every verification point, not a demand-filled cache
- ADR-04 · Invalid answers are constant in shape and cost, from a closed set of reasons

**Revocation and propagation**: How a key stops working everywhere, and how quickly that can be proved.

- ADR-05 · Revocation is an independent, append-only, quorum-acknowledged log
- ADR-06 · Push feed with a poll fallback and a heartbeat, not a probabilistic filter
- ADR-07 · Staleness is published on every response, and bounded by a scope-graded fail-closed ceiling
- ADR-08 · Bulk revocation is a tenant epoch bump, not N revocations

**Cryptography and custody**: What is stored, what is not, and what a stolen database is worth.

- ADR-09 · A fast keyed digest with an HSM-held pepper, not a memory-hard KDF
- ADR-10 · The pepper is versioned, so a compromise is a rotation and not a re-issuance
- ADR-11 · The secret is returned once and is unrecoverable by construction

**Scope and the authorization contract**: What a credential carries, who may widen it, and what the platform refuses to decide.

- ADR-12 · The service reports what a credential carries and never what it permits
- ADR-13 · A flat, additive, centrally owned scope vocabulary
- ADR-14 · No principal may issue a credential wider than itself, checked at issuance
- ADR-15 · last_used_at is an observation, never an authorization input

**Detection and response**: Finding a leaked credential, and what the platform is allowed to do about it on its own.

- ADR-16 · Greppable prefixes and a partner endpoint that returns one bit
- ADR-17 · Auto-revoke on a confirmed leak; anomaly-only response is opt-in and reversible

**Platform, deployment and operations**: Where it runs, how a change reaches it, and how its central claim is kept honest.

- ADR-18 · Verification is active-active everywhere; the control plane has one leader
- ADR-19 · A revocation canary, run from a project the platform's operators cannot deploy to
- ADR-20 · Config artefacts are immutable, ring-rolled, and refused rather than fallen back from

## Technology by capability

Every capability and what it is realised with, the origin of that choice, the alternative that was weighed, and the record that argues it. Where the requirement stays vendor-neutral, this table is where the stack commits.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Credential record store | Cloud Spanner, multi-region nam-eur-asia1, tenant-partitioned | Google Cloud | Regional PostgreSQL with logical replication | Strong consistency across regions with no application-level conflict handling; the same instance carries the revocation log, whose RPO 0 is the requirement that justifies the cost. | ADR-05 |
| Revocation log | Append-only Spanner table with a monotonic sequence and quorum commit | This design | A status column on the credential record | Revocation needs its own durability and ordering, independent of credential replication and immune to a restore. | ADR-05 |
| Revocation distribution | Pub/Sub global topic, 5-second poll fallback, 10-second heartbeat | Google Cloud | Cuckoo filter distributed to every point | Push latency with poll self-healing, and a positive liveness signal without which a dead channel is indistinguishable from a quiet one. | ADR-06 |
| Verification point | Envoy ext_authz sidecar plus Go and Java libraries | Open source | Central verification service behind a load balancer | Verification must cost a memory lookup rather than a network hop, or callers will cache it badly and skip it under load. | ADR-02 |
| Projection warm-up | Regional Cloud Storage snapshot every 60 s, replayed forward from its sequence | This design | Demand-filled cache reading Spanner on a miss | Removes the cold path entirely, so a fleet restart is a download rather than a load spike against the credential store. | ADR-03 |
| Shared projection for thin clients | Memorystore for Redis, per region | Google Cloud | Each thin client holding its own projection | A bounded escape hatch for callers that genuinely cannot hold state, with its own p99 ≤ 15 ms budget and a reported call rate. | ADR-02 |
| Digest and pepper custody | HMAC-SHA-256 under a Cloud HSM pepper, versioned | Google Cloud | Argon2id over the secret | 160 bits of platform entropy is not a password; the defence is custody of the pepper, and the digest must be affordable 250,000 times a second. | ADR-09 |
| Envelope encryption | Cloud KMS CMEK per tenant over Spanner and BigQuery | Google Cloud | Provider-managed default encryption | Tenant-scoped key hierarchies make a customer's crypto-shred a real operation rather than a promise. | ADR-11 |
| Usage counters | Cloud Bigtable, asynchronous aggregation | Google Cloud | Counters in the credential store | last_used_at is an observation, so it may be lossy and lagging — which is exactly what keeps it off the hot path. | ADR-15 |
| Audit store | BigQuery hot tier, hash-chained records, tiered to locked Cloud Storage | Google Cloud | A write-once ledger service | Tamper evidence from per-tenant chaining plus object-lock retention, with query cost visible to whoever runs the query. | ADR-12 |
| Console and management API | GKE behind a global load balancer, IAP for the console, Cloud Armor at the edge | Google Cloud | Cloud Run for both | The control plane and the sidecar share a runtime and a release pipeline, which keeps one projection implementation rather than two. | ADR-18 |
| Identity | Platform IdP over OIDC for humans, GKE workload identity for machines | Google Cloud | Service account keys | A credential service that needed a long-lived bootstrap credential of its own would not have solved its own problem. | ADR-14 |
| Leak detection intake | Unauthenticated partner endpoint behind Cloud Armor, hard per-partner quota | This design | Authenticated partner integration with a shared secret | Partners cannot hold a platform credential per customer; the endpoint returns one bit and structurally cannot leak anything else. | ADR-16 |
| Config and vocabulary delivery | Signed immutable artefacts, ring rollout, last-good pinning | This design | Live config pushed from a database | A poison config reaches every verification point at once, which makes it the fastest available way to take the platform down. | ADR-20 |
| Assurance | Revocation canary in a separate project and region, dead-man alerting | This design | Propagation metrics emitted by the platform itself | A platform cannot be the only judge of whether it can still revoke; the canary is the only signal that catches a silent propagation failure. | ADR-19 |

## The decisions, and the alternatives that lost

### Credential form and verification

*What a credential is, and what it costs to check one.*

#### ADR-01 · The credential is an opaque pointer, not a self-contained claim

**Status:** Accepted  ·  **Shown on views:** 01, 02, 06, 12, 20

*Does a long-lived developer credential carry its own scopes and expiry, verified by a signature, or is it an opaque handle that must be looked up?*

**Context.** A self-contained signed token verifies with no state at all: the verifier holds a public key and reads the claims out of the token. That is the right answer inside a trust boundary for a credential that lives ninety seconds. A developer API key lives eighteen months, sits in a CI variable, and its defining event is the day it appears in a public commit. On that day the platform is asked one question: how long does it stay useful to whoever found it. A self-contained token's honest answer is "until it expires", and the usual remedy — a denylist consulted on every verification — reintroduces exactly the state the token format existed to avoid, with none of the design attention that state would otherwise have received.

**Decision.** Long-lived credentials are opaque: a public prefix encoding class and environment, a key identifier usable for lookup and logging, and a high-entropy secret. Nothing about what the credential may do travels inside it. Authority is resolved by lookup, always. Short-lived self-contained tokens are permitted in Phase 2 but only as something minted from an opaque credential, which moves the revocation problem to the exchange rather than removing it.

**How it is realised on Google Cloud.** Format `sk_live_<kid>_<secret>`, with the class and environment in the prefix so a scanner, a log line and an error message can all identify it without the secret. The credential record lives in Cloud Spanner, tenant-partitioned. Nothing signs anything: there is no key pair, no JWKS endpoint, and no verifier that could honour a credential on the strength of the credential alone.

| Option | Verdict | Reasoning |
|---|---|---|
| Opaque handle with lookup | Chosen | Revocation is immediate and total, and the credential discloses nothing about the tenant. Costs state at verification time, which ADR-02 and ADR-03 are about. |
| Signed self-contained token (JWT-style) | Rejected | Stateless verification, but valid until expiry whatever anyone does. A denylist to fix that is the lookup you were avoiding, built late and designed less. |
| Hybrid: short-lived tokens minted from a long-lived key | Deferred | Right for machine-to-machine traffic and planned for Phase 2. Does not remove the problem: the long-lived key at the exchange is still this decision. |
| Signed token with a very short expiry and no revocation | Right elsewhere | Correct inside a service mesh where credentials live for seconds and the issuer is on the same network. Not correct for a string a customer pastes into a config file. |

**What it buys**

- Revocation becomes possible at all, which is the whole product.
- The credential leaks nothing: a key in a public commit does not disclose the tenant, the scopes or the environment beyond what the prefix says deliberately.
- Scope changes take effect on the next verification, without waiting for a token to expire.

**What it costs**

- Every verification needs state, which forces the projection, the feed and the staleness machinery — most of this architecture.
- A verification point that has never been warmed cannot verify anything, where a signature check would have worked from the first request.

**Choose differently when.** Choose a self-contained token when credentials live minutes rather than months, when the verifier cannot hold state at all, or when there is no requirement to revoke faster than expiry. A webhook signature and a service-mesh identity are both in that category.

**Why it holds up over time.** This decision depends on no technology. Spanner, Pub/Sub, Google Cloud and HMAC can all be replaced without touching it. What would break it is a future team adding a 'fast path' that reads scopes out of the credential to avoid a lookup — which is why the refused crossing is drawn explicitly in view 20 rather than left as a convention.

> **Lesson.** A credential's format decides what you can do on your worst day. Choose it for the incident, not for the benchmark.

#### ADR-02 · Verification is a local read, shipped as a library and a sidecar

**Status:** Accepted  ·  **Shown on views:** 02, 06, 08, 12

*Where does verification execute — in the calling process, in a sidecar beside it, or in a central service the caller calls?*

**Context.** ADR-01 made verification stateful. The naive consequence is a central verification service, which is the most common design and the one that fails predictably: at 250,000 requests per second it is a network hop on every request in the platform, it becomes the single most critical dependency anything has, and the teams calling it respond exactly as you would expect — they cache the answer for five minutes with no revocation awareness, or they skip the check under load. A verification that is expensive gets skipped, and a verification that gets skipped is worse than one that never existed, because nobody knows which requests were checked.

**Decision.** Verification executes locally at the calling service: a library in-process where the language allows, an Envoy `ext_authz` sidecar otherwise. The verification point holds a projection of active credentials and a revocation overlay, and answers from memory. A remote verification API exists for thin clients that genuinely cannot hold state, is bounded at p99 ≤ 15 ms, and is explicitly not the intended path.

**How it is realised on Google Cloud.** Sidecar container in each GKE pod, exposing `ext_authz` over localhost to the Envoy in front of the application; a Go and a Java library for the services that would rather link it. Both share one projection implementation and one revocation-feed client, so there is one staleness behaviour to reason about and one to test. The remote API runs in each region backed by a Memorystore-held projection, and its call rate is a reported metric because a rise in it means a team could not adopt the library.

| Option | Verdict | Reasoning |
|---|---|---|
| Local library plus sidecar | Chosen | No network hop, no shared dependency, and nobody is tempted to cache the answer because the answer is already local. |
| Central verification service | Rejected | One hop per request platform-wide, and the most critical dependency in the estate. Reliably produces badly-cached client-side workarounds that outlive the incident that caused them. |
| Gateway-only verification | Rejected | Cheap, but only the edge is checked. Internal replay, async workers and service-to-service calls then trust an upstream assertion nobody re-verifies. |
| Verification in the data store's own authorization layer | Right elsewhere | Elegant when every access goes through one store. Not available to a platform with 180 services and several storage technologies. |

**What it buys**

- Verification costs a memory lookup, which makes 'verify on every request' a decision nobody has to defend in a budget review.
- No shared runtime dependency: a control-plane outage does not stop a single request being served.
- One implementation of the staleness rule, rather than one per calling team.

**What it costs**

- Memory in every pod, proportional to the number of active credentials — the assumption ADR-03 is about.
- A library is a product: versions, upgrades, a deprecation path and a support burden the platform team now owns.
- The remote API remains as an escape hatch and will be used by someone; it needs its own capacity and its own SLO.

**Choose differently when.** Choose a central service when the calling fleet is small and homogeneous, when credentials number in the thousands rather than the millions, or when the per-request latency budget is generous enough that a hop is invisible. Choose gateway-only when the platform genuinely has one ingress and no internal re-entry.

**Why it holds up over time.** The sidecar mechanism will age — service meshes come and go — but the property that matters is that verification is co-located with the caller. Whatever replaces Envoy in ten years will still be a place to put this, and the contract (a decision plus a staleness figure) is unchanged by the move.

> **Lesson.** A check that is expensive will be skipped, and you will not be told. Make the correct path the cheap one.

#### ADR-03 · A full projection at every verification point, not a demand-filled cache

**Status:** Accepted  ·  **Shown on views:** 08, 10, 16, 22

*Does a verification point hold every active credential, or only the ones it has seen?*

**Context.** A demand-filled cache is smaller and obviously correct in steady state. Its problem is the cold path: on a miss it must read the durable store. That path runs precisely when it is most dangerous — a fleet restart, a deployment, a region rejoining after a partition — and turns a rollout into a load spike against the one store that must not be slow. It also makes the first request for each credential slower than the rest, which is a timing signal about which keys are in use. A full projection removes the cold path entirely, at the cost of memory proportional to the whole platform at every point.

**Decision.** Each verification point holds a projection of all active credentials for the tenants it serves, warmed from a regional snapshot at start-up, with admission gated on readiness. There is no read from the credential store on the request path, and such a read is counted and alerted on as an exception rather than budgeted as a cost.

**How it is realised on Google Cloud.** The snapshot builder writes a regional object to Cloud Storage every 60 seconds, holding key identifier, digest, pepper version, scope set, constraints and expiry for every active credential. A starting verification point loads the snapshot, then replays the revocation feed forward from the snapshot's sequence number before it reports ready. At the assumed 2.5 million credentials and roughly 160 bytes per entry this is about 400 MB per point, which fits the platform's standard pod but not its smallest.

| Option | Verdict | Reasoning |
|---|---|---|
| Full projection, snapshot-warmed | Chosen | No cold path at any point in the lifecycle, and constant-cost answers for hits and misses alike. |
| Demand-filled cache with a TTL | Rejected | Small, but the cold path exists exactly when the store is least able to serve it, and the TTL is a second, undeclared staleness mechanism. |
| Partitioned projection by key-prefix shard | Deferred | Keeps the no-cold-path property with a fraction of the memory, at the cost of routing. This is the fallback if the memory measurement in the prototype fails. |
| Bloom filter of valid identifiers plus a store read on a hit | Rejected | Cheap negative answers, but every legitimate request still reads the store — the cold path made permanent rather than removed. |

**What it buys**

- A restart storm is a snapshot-download event rather than a thundering herd against Spanner.
- Hits and misses cost the same, so response timing discloses nothing about which identifiers exist.
- A partitioned region keeps verifying from what it has, with its staleness stated.

**What it costs**

- About 400 MB of memory in every pod running a sidecar — a real cost, multiplied by the fleet.
- The snapshot path becomes a dependency of start-up, and a stale or corrupt snapshot is a new failure mode.
- Credential growth is now a capacity planning input for every team on the platform, not just for this one.

**Choose differently when.** Choose a demand-filled cache when credentials number in the tens of thousands, when the working set is a small fraction of the whole, or when verification points are numerous and small. Choose the partitioned projection as soon as memory per point exceeds what the smallest deployable unit can carry.

**Why it holds up over time.** The number that decides this — active credentials times bytes per entry, against pod memory — moves in both directions over a decade. The decision is therefore written with its trigger attached, and the partitioned fallback is designed rather than hypothetical, so crossing the threshold is a planned migration rather than an incident.

> **Lesson.** Caches fail on the cold path, and the cold path arrives at the worst moment by construction. Price it before choosing the smaller option.

#### ADR-04 · Invalid answers are constant in shape and cost, from a closed set of reasons

**Status:** Accepted  ·  **Shown on views:** 08, 12, 20

*What does verification return when a credential is unknown, revoked, expired or constrained out — and how much does the answer disclose?*

**Context.** The obvious implementation returns quickly for an unknown identifier and more slowly for a known one, because the second does more work. That difference is an oracle: an attacker with a partial key can learn which identifiers exist by timing. The same applies to error text — 'unknown key' versus 'key revoked' distinguishes a guess from a hit. Meanwhile the operator needs to tell those cases apart to answer a support ticket, so the information cannot simply be discarded.

**Decision.** Verification returns VALID with an authorization context, or INVALID with one of a closed set: NOT_FOUND, REVOKED, EXPIRED, SUSPENDED, CONSTRAINT_FAILED, TENANT_DISABLED. The reason is recorded in telemetry and available to the operator and to the owning tenant; it is not returned to an unauthenticated caller, which receives a single undifferentiated failure. The work done on the invalid path is padded to the shape of the valid path so the two do not differ in timing.

**How it is realised on Google Cloud.** The sidecar computes the digest and performs the overlay check regardless of whether the identifier resolves, discarding the result on a miss. The API response carries an opaque failure and a request identifier; the reason is emitted at 100% sampling to the telemetry path and surfaced in the console against the credential when the caller can be attributed to a tenant.

| Option | Verdict | Reasoning |
|---|---|---|
| Closed reason set, disclosed only to the tenant | Chosen | The operator and the owner can debug; an outsider learns one bit. The padding closes the timing channel that the reason set would otherwise reopen. |
| Detailed reasons returned to the caller | Rejected | Excellent developer experience and a working enumeration oracle. The developer experience is recovered through the console, where the caller is authenticated. |
| A single opaque failure, recorded nowhere | Rejected | Discloses nothing and makes every support conversation guesswork, which pushes people toward wider keys and longer expiries. |
| Rate-limit the caller instead of padding | Right elsewhere | Sufficient where the attacker cannot distribute the guessing. Here both are applied, but padding is what holds when the source is a botnet. |

**What it buys**

- Response timing and content disclose nothing about which key identifiers exist.
- Support and self-service debugging keep the full reason, through an authenticated path.
- Denial reasons are complete in telemetry, which is what makes the anomaly signals in ADR-17 possible.

**What it costs**

- The invalid path does work it does not need to do, which costs CPU on exactly the requests you would rather drop cheaply.
- Padding is a property that must be tested, and it will regress silently the first time someone optimises the miss path.

**Choose differently when.** Return detailed reasons directly when the caller is always authenticated by a second factor before reaching verification, or when the credential space is small enough that enumeration is a non-threat.

**Why it holds up over time.** Timing side channels do not go away with better hardware; if anything, more predictable execution makes them easier to exploit. The rule — equal work on both paths — is stable, and the test for it is a benchmark that can live in CI for a decade.

> **Lesson.** An error message is an interface to an attacker as well as a developer. Route the detail through authentication rather than deleting it.

### Revocation and propagation

*How a key stops working everywhere, and how quickly that can be proved.*

#### ADR-05 · Revocation is an independent, append-only, quorum-acknowledged log

**Status:** Accepted  ·  **Shown on views:** 02, 06, 10, 14, 16

*Is a revocation a status column on the credential record, or a separate structure with its own durability guarantee?*

**Context.** Revoking by updating a status column is simpler and wrong in one specific way: the revocation now inherits the credential store's replication behaviour and its recovery objective. A restore from a backup taken before the revocation silently un-revokes the key. A cross-region replication lag delays the revocation exactly as it delays everything else. And a bulk revocation becomes a hundred thousand row updates propagated as a hundred thousand changes. The asymmetry is the point: losing an issuance is recoverable, because the customer can issue another. Losing a revocation is not recoverable, because the credential is already in a public commit.

**Decision.** Revocation state is an independent append-only log with a monotonic sequence, replicated and evaluated separately from credential records, carrying RPO 0. A revocation is acknowledged to the caller only after a durable multi-region quorum write. Once acknowledged it can never be undone — not by replication catch-up, not by cache repopulation, not by a restore from backup.

**How it is realised on Google Cloud.** A Spanner table in the multi-region `nam-eur-asia1` configuration, written with a commit that spans regions before the API returns, and read by the feed publisher through a change stream. Restores of the credential-record tables are always accompanied by a replay of the revocation log forward from the restore point, enforced in the restore runbook and exercised in the prototype scenarios.

| Option | Verdict | Reasoning |
|---|---|---|
| Independent append-only log, quorum-acknowledged | Chosen | Revocation gets its own durability, its own ordering and its own propagation, none of which are hostage to the credential store's behaviour. |
| Status column on the credential record | Rejected | Simple, and a restore un-revokes keys. The failure is silent and arrives during an incident, which is the worst combination available. |
| Tombstone rows with a TTL | Rejected | Bounded storage, and the revocation expires. A credential whose revocation has aged out is live again, which is precisely the class of bug this service exists to prevent. |
| Distributed consensus group per region | Right elsewhere | Right when there is no managed multi-region store available. Here it is operational burden the platform can buy its way out of. |

**What it buys**

- An accepted revocation survives the loss of a region and the restore of any store.
- Revocation ordering is independent of credential replication, so propagation latency is a property this team controls.
- Bulk revocation has somewhere to be cheap — see ADR-08.

**What it costs**

- A multi-region quorum write puts roughly 150 ms into the revoke API call, which is invisible to a human and would matter if revocation were a high-rate operation. It is not.
- A second store to operate, back up, monitor and reason about.
- Every restore procedure in the platform now has an extra mandatory step, which is exactly the kind of step that gets skipped unless it is tested.

**Choose differently when.** Use a status column when revocation is rare, non-urgent and the store is single-region anyway — an internal tool with ten users. The moment revocation is an incident-response action, it needs its own durability.

**Why it holds up over time.** The claim 'an accepted revocation is never lost' is a contract, not an implementation. Spanner can be replaced by whatever offers a multi-region quorum commit in ten years, and the restore-plus-replay rule transfers unchanged. What keeps it honest is the prototype scenario that restores a pre-revocation backup and checks the key is still dead.

> **Lesson.** Not all writes deserve the same durability. Find the one whose loss you cannot undo, and give it its own guarantee.

#### ADR-06 · Push feed with a poll fallback and a heartbeat, not a probabilistic filter

**Status:** Accepted  ·  **Shown on views:** 06, 08, 14, 18, 22

*How does a revocation reach thousands of independent verification points, and how is the mechanism's own failure detected?*

**Context.** Three mechanisms are available and each fails differently. A push bus is fast and fails silently: a subscriber that stops receiving looks identical to a quiet period. A pull feed is self-healing — a point that misses a poll simply catches up — but adds a floor to propagation equal to the poll interval. A probabilistic structure such as a cuckoo filter of revoked identifiers is small enough to distribute anywhere, and its false positives here mean denying a valid credential, which is a customer outage caused by a data structure's sizing.

**Decision.** Revocations are pushed over Pub/Sub for latency, with a 5-second poll fallback for self-healing, and a heartbeat on the same channel so that silence is distinguishable from quiet. A verification point tracks the highest sequence number it has applied; falling behind is a measured quantity, not an unknown. The probabilistic filter is rejected for the primary path and reconsidered only above roughly ten thousand verification points.

**How it is realised on Google Cloud.** A global Pub/Sub topic carrying ordered revocation records, each with its sequence number; subscribers apply them additively to the overlay. A separate poll endpoint returns everything after a given sequence, called every 5 seconds regardless of push traffic. The heartbeat is a record on the same topic emitted every 10 seconds whether or not there are revocations, so its absence is unambiguous.

| Option | Verdict | Reasoning |
|---|---|---|
| Push plus poll plus heartbeat | Chosen | Push latency, poll self-healing, and the one thing neither gives alone — a positive signal that the channel is alive. |
| Push only | Rejected | Fastest, and its failure mode is a verification point that honours a revoked credential forever while reporting perfect health. |
| Poll only | Rejected | Simplest and most robust, but the poll interval becomes the propagation floor. At the 2-second p50 target the polling rate would itself be a load problem. |
| Cuckoo filter distributed to every point | Deferred | Tiny and attractive at very large fleets. A false positive denies a valid credential, so it needs a confirmation path — which is a store read, and the cold path returns. |

**What it buys**

- Propagation p50 tracks push latency while recovery from a missed message needs no intervention.
- A dead channel is detected in seconds by every point independently, which is what makes the staleness rule in ADR-07 enforceable.
- Sequence numbers make 'how far behind is this point' a number rather than a guess.

**What it costs**

- Two delivery paths to build, test and keep consistent, plus the heartbeat as a third thing that can break.
- Poll traffic at 5-second intervals from every verification point is a constant baseline load the feed must serve even when nothing is happening.

**Choose differently when.** Poll-only is right when the propagation target is measured in minutes rather than seconds. The filter becomes right when the number of verification points makes per-point state untenable and a confirmation round trip is affordable on the small fraction of requests it affects.

**Why it holds up over time.** Message buses will change. What survives is the three-part structure: a fast path, a self-healing path, and a positive liveness signal. A design that has all three can swap any one of them; a design that has only the first cannot tell when it has stopped working.

> **Lesson.** Every asynchronous channel needs a heartbeat, because silence and health are otherwise the same observation.

#### ADR-07 · Staleness is published on every response, and bounded by a scope-graded fail-closed ceiling

**Status:** Accepted  ·  **Shown on views:** 02, 08, 12, 14, 22

*What does a verification point do when it can no longer prove that its projection is fresh?*

**Context.** This is the decision that determines whether the architecture is honest. A point that serves from an old projection indefinitely will eventually honour a revoked credential, and will do so while reporting success. Failing closed protects against that and takes a customer's integration down during an outage the customer did not cause — the platform converting its own incident into theirs. Failing open does the reverse and makes the revocation SLO a fiction the moment the feed breaks. Neither pure answer is defensible for a platform whose credentials authorise both reading an invoice and issuing a refund.

**Decision.** Every verification response carries the projection's staleness. Past 120 seconds without a successful feed poll, a verification point refuses credentials for write- and admin-graded scopes while continuing to serve read-graded scopes, according to per-tenant policy. The grading comes from the scope vocabulary, so the behaviour is derived from the scope's declared grade rather than from a list maintained separately.

**How it is realised on Google Cloud.** The staleness guard runs inside the sidecar, comparing wall-clock time against the last successful poll and the last applied sequence number. The scope grade travels in the snapshot and the feed, so the guard needs no additional lookup. The value is returned in the authorization context and exported as a histogram, so the fleet-wide staleness distribution is visible rather than inferred.

| Option | Verdict | Reasoning |
|---|---|---|
| Scope-graded fail-closed at a declared ceiling | Chosen | Refuses what a stale projection could actually get wrong in a costly way, while an outage does not stop customers reading their own data. |
| Fail closed entirely | Rejected | Simplest to reason about and to test. A feed outage becomes a total platform outage, which is a price the business will refuse after the first occurrence — and then fail open badly. |
| Fail open past the ceiling | Rejected | Available, and makes the revocation SLO untrue exactly when it is being relied on. The failure is invisible, which is what disqualifies it. |
| No ceiling — serve whatever is held | Rejected | The default behaviour of every cache, and the reason a revocation becomes a press release. |

**What it buys**

- Staleness is a number in the response rather than an unstated property, so callers can make their own decisions with it.
- The worst outcome of a propagation failure is bounded and known in advance rather than discovered.
- Read traffic — the majority — survives a feed outage.

**What it costs**

- Two behaviours to specify, implement and test per scope grade, and a grading that must be right in the vocabulary.
- A feed outage still refuses writes platform-wide, which will be an incident with customer impact and a difficult conversation.
- The 120-second ceiling is an invented number and the one most likely to be argued down under pressure.

**Choose differently when.** Fail closed entirely when every scope is consequential — a payments-only platform. Fail open when credentials authorise nothing that cannot be undone and availability is the dominant concern, and say so explicitly rather than arriving there by omission.

**Why it holds up over time.** The ceiling's value will change with measured propagation data; the structure will not. Because staleness is on the response, any future propagation mechanism can be evaluated against the same contract, and a regression shows up as a distribution shift rather than as an incident.

> **Lesson.** Every cache has a staleness. The only question is whether the system knows it, says it, and changes behaviour at a declared bound.

#### ADR-08 · Bulk revocation is a tenant epoch bump, not N revocations

**Status:** Accepted  ·  **Shown on views:** 10, 11, 14, 15

*What happens when a customer needs every credential issued before a certain moment to stop working at once?*

**Context.** This is the request after a laptop is stolen, a contractor leaves, or a repository is found to have been public for a year. Done naively it is a hundred thousand individual revocations: a hundred thousand log records, a hundred thousand feed messages, and a propagation time that scales with the size of the incident — slowest exactly when it matters most. It also makes the feed's worst-case load a function of an event nobody controls.

**Decision.** Each tenant carries a monotonic revocation epoch. A bulk revocation increments it, in one ordered write, and every credential issued before that epoch is invalid from that moment. Propagation cost is one message regardless of whether the bump kills one credential or a million. Individual revocations remain per-credential records; the epoch handles the set case.

**How it is realised on Google Cloud.** `tenant.revocation_epoch` in Spanner, written in the same transaction that appends the epoch record to the revocation log. Each credential record carries its issuance epoch; the verification check is a single integer comparison alongside the overlay lookup, adding no measurable cost to the hot path. Bulk operations scoped more narrowly than a whole tenant — by principal, by credential class, by creation window — are expressed as a filtered epoch, evaluated the same way.

| Option | Verdict | Reasoning |
|---|---|---|
| Tenant epoch with a comparison at verification | Chosen | Constant propagation cost, constant verification cost, and the incident's size stops being the feed's problem. |
| N individual revocation records | Rejected | Uniform and simple, with propagation time proportional to incident size. The worst case is unbounded and arrives during a security incident. |
| Disable the tenant entirely | Rejected | Achieves containment by taking the customer offline, which is a different product decision wearing a security costume. |
| Batch revocations into chunked messages | Right elsewhere | Reasonable when there is no per-credential epoch to compare against. Reduces the constant but keeps propagation proportional to the set. |

**What it buys**

- 100,000 credentials revoked and propagated within the same window as one — the requirement's bulk target is met by construction rather than by capacity.
- The feed's worst-case load is independent of incident size.
- Narrower bulk operations reuse one mechanism and one code path.

**What it costs**

- Every credential record must carry its issuance epoch, and every verification must compare it — a small permanent cost on the hot path.
- An epoch bump is blunt: a credential issued a second before the cut dies with the rest, and the customer must re-issue it.
- Reasoning about 'why did this key stop working' now has two mechanisms to check rather than one.

**Choose differently when.** Skip the epoch when bulk revocation is genuinely rare and small — a few hundred credentials at a time — and the propagation target has room for it. The mechanism is cheap but not free, and it complicates the explanation of why a credential failed.

**Why it holds up over time.** Epochs are a decades-old technique precisely because they are independent of the storage and transport underneath. As the platform grows, the epoch's advantage grows with it: the naive design gets worse with scale and this one does not.

> **Lesson.** When the incident-response operation is the one that must be fastest, make its cost independent of the incident's size.

### Cryptography and custody

*What is stored, what is not, and what a stolen database is worth.*

#### ADR-09 · A fast keyed digest with an HSM-held pepper, not a memory-hard KDF

**Status:** Accepted  ·  **Shown on views:** 06, 10, 11, 12, 13, 20

*How is a credential secret stored so that a stolen database is worthless, without making verification unaffordable?*

**Context.** The standard answer for storing a secret is a memory-hard key derivation function — Argon2id or scrypt — deliberately slow to make offline cracking expensive. That answer is correct for passwords, which are low-entropy and chosen by humans. It is wrong here for two reasons. A platform-generated secret carries 160 bits of entropy from a CSPRNG, which is not brute-forceable regardless of how fast the digest is; the slowness buys nothing against that input. And verification happens 250,000 times a second on the request path, where a deliberately expensive function is not merely costly but architecture-breaking.

**Decision.** Credentials are stored as HMAC-SHA-256 of the secret under a pepper held in Cloud HSM, with the pepper version recorded alongside. The security against an offline attack comes from the pepper being outside the database rather than from the digest being slow. Secrets are required to carry at least 160 bits of entropy, which is what makes this sound — the entropy requirement and the digest choice are one decision, not two.

**How it is realised on Google Cloud.** Cloud KMS with a hardware protection level holds the pepper; issuance calls KMS to compute the digest. Verification points cache the pepper by version in memory, fetched over workload-identity-authenticated calls at start-up and on version change, which is why KMS being unreachable stops issuance but not verification. The pepper is never written to Spanner, to a snapshot, to a backup or to a log.

| Option | Verdict | Reasoning |
|---|---|---|
| HMAC-SHA-256 with an HSM-held pepper | Chosen | Microseconds per verification, and a database dump alone yields nothing. Rests entirely on the pepper being genuinely isolated and on the entropy requirement holding. |
| Argon2id or scrypt | Rejected | The right answer for passwords. Here it defends against an attack the entropy already prevents, at a cost that makes local verification impossible. |
| Plain SHA-256, no pepper | Rejected | Fast, and a stolen database plus a rainbow table over the key format is a real attack once entropy assumptions slip even slightly. |
| Split scheme — fast digest plus a slow proof-of-possession tier | Deferred | Worth revisiting if customer-chosen credential material is ever accepted, which would break the entropy assumption this decision depends on. |

**What it buys**

- Verification costs microseconds, which is what makes ADR-02 possible at all.
- Exfiltration of the credential store alone does not permit offline forgery or cracking.
- The pepper's separate custody gives ADR-10 its rotation story.

**What it costs**

- Security depends on the pepper's isolation. Whatever compromises the database must not also reach KMS, which is a claim about the deployment, not about the cryptography.
- The 160-bit entropy requirement is load-bearing and invisible in the digest code; a future change accepting customer-supplied key material would silently invalidate this decision.
- KMS becomes an issuance-path dependency with its own availability characteristics.

**Choose differently when.** Use a memory-hard KDF the moment any credential material is chosen by a human or supplied by a customer. Use a plain fast hash only where there is no store to steal.

**Why it holds up over time.** SHA-256 and HMAC have a long remaining life, and the construction is agile: the pepper version field already carries an algorithm identifier, so moving to a different primitive is the same operation as rotating a pepper. The decision's real dependency is the entropy requirement, which is why it is written into this record rather than left in an issuance function.

> **Lesson.** Match the defence to the input. Slow hashing protects weak secrets; strong secrets need custody instead.

#### ADR-10 · The pepper is versioned, so a compromise is a rotation and not a re-issuance

**Status:** Accepted  ·  **Shown on views:** 10, 11, 12, 16, 22

*What actually happens on the day the credential store is exfiltrated?*

**Context.** ADR-09 makes a stolen database worthless while the pepper stays secret. The question that follows is what the response looks like if the pepper itself is suspected — or simply when a scheduled rotation comes due. Without versioning, changing the pepper invalidates every digest in the store, which means asking every customer to re-issue every credential: an event no platform survives gracefully and therefore an event that never happens, which means the pepper is never rotated, which means the scheduled rotation is a policy nobody has ever executed.

**Decision.** Every credential record carries the pepper version used to compute its digest. Verification selects the pepper by the record's version, so several versions are valid at once. Rotation issues a new version, digests new credentials under it, and migrates existing records opportunistically — on next successful verification, where the secret is momentarily in hand — with a bounded campaign for the remainder. Annual rotation is scheduled and, critically, exercised.

**How it is realised on Google Cloud.** Cloud KMS key versions map one-to-one to pepper versions; verification points cache the small set of live versions. On a successful verification under an old version, the sidecar reports the identifier to the control plane, which re-digests the secret under the current version within the same request. Credentials never verified during the campaign are re-digested at next use or expire; the version distribution is a reported metric, so a stalled migration is visible.

| Option | Verdict | Reasoning |
|---|---|---|
| Versioned pepper with opportunistic migration | Chosen | Rotation without customer impact, which is what makes it something the team will actually do rather than write down. |
| Single pepper, rotation by mass re-issuance | Rejected | Simple and never executed. A rotation procedure nobody can afford to run is equivalent to no rotation procedure. |
| Per-tenant peppers | Deferred | Smaller blast radius per compromise, and a KMS call pattern and cache footprint that grow with tenant count. Revisit if a tenant ever requires cryptographic isolation by contract. |
| Re-encrypt digests in place with a new pepper | Rejected | Not possible: an HMAC is not re-keyable without the input, and the input is the secret, which the platform does not have. |

**What it buys**

- Store compromise is answered by rotating one key, not by a platform-wide customer event.
- Rotation is routine, so the procedure is exercised and known to work when it is needed urgently.
- Old versions can be retired on a schedule with visible progress rather than by assumption.

**What it costs**

- Verification points hold several pepper versions, and the migration campaign is a long tail that needs owning.
- A credential never used again keeps its old digest until expiry, so retiring a version fully takes as long as the longest credential lifetime.
- The opportunistic re-digest path touches the secret during verification, which is a code path that must be as carefully reviewed as issuance.

**Choose differently when.** A single pepper is acceptable when the credential population is small enough that mass re-issuance is a real option, or when credentials are short-lived enough that rotation happens naturally within a month.

**Why it holds up over time.** Versioning is what turns any cryptographic choice into a reversible one. Its value increases over a decade: the primitive chosen today will be questioned, and the version field is the mechanism by which that question has a cheap answer.

> **Lesson.** A rotation procedure that is expensive is a rotation procedure that does not exist. Design the migration before choosing the key.

#### ADR-11 · The secret is returned once and is unrecoverable by construction

**Status:** Accepted  ·  **Shown on views:** 04, 10, 13, 19

*Can the platform ever show a customer their credential secret again?*

**Context.** Every support organisation eventually asks for this. A developer has lost the secret, their deployment is broken, and re-creating the key means touching production configuration. Storing the secret encrypted — recoverable under a KMS key, behind a strong access control — solves that conversation and creates a far worse one: there is now a path, however well guarded, by which the platform can produce a customer's live credential. That path will be used by an insider, exposed by a misconfiguration, or subpoenaed. It also destroys the claim the platform makes to its customers about what it holds.

**Decision.** The secret is generated, digested and returned inside a single request, and no recoverable copy exists in any store, backup, log, export or support tool. A restore of every store the platform operates cannot revive one secret. The lost-secret problem is answered by rotation — issue a successor, overlap, retire the predecessor — rather than by recovery.

**How it is realised on Google Cloud.** The secret exists in the issuance service's memory for the duration of one response. Logging middleware redacts anything matching the credential format at the emitter, not at the sink. The console shows the secret once behind an explicit affordance with a one-click copy, and thereafter only the prefix and identifier. Envelope encryption with per-tenant CMEK covers the records that do exist, which are digests.

| Option | Verdict | Reasoning |
|---|---|---|
| Show once, never store | Chosen | A structural guarantee rather than a policy one. Costs one difficult moment in the developer journey, recovered by making rotation trivial. |
| Store encrypted, recoverable under strong controls | Rejected | Solves a real support problem by creating a path to every customer's live credentials. The controls are the only thing standing between an insider and the platform's worst day. |
| Escrow with the customer's own KMS key | Deferred | Moves custody to the customer, which is defensible. Worth revisiting only if a regulated customer requires escrow by contract, and it should be per-tenant and visible. |
| Show once, with a 5-minute re-display window | Rejected | A small window is still a stored secret with a TTL, and the TTL is the only thing preventing the attack. |

**What it buys**

- The platform can state truthfully that it cannot reproduce a customer's secret, and that statement survives an insider and a backup misconfiguration.
- Backups, exports and support tooling are categorically out of the secret's blast radius.
- There is no recovery endpoint to attack, rate-limit or audit.

**What it costs**

- A lost secret is a rotation, which touches production configuration. This is the trough in the developer journey and it is deliberate.
- Redaction must be enforced at every emitter, and one careless log line in one service reintroduces the risk this decision removes.

**Choose differently when.** Escrow becomes right when the customer, not the platform, holds the key, and the customer has asked for it in writing. It is never right as a convenience.

**Why it holds up over time.** Structural guarantees outlive organisational ones. Ten years and four teams later, nobody can quietly relax this: making secrets recoverable would require changing the issuance path, which is a visible and reviewable act rather than a permission change.

> **Lesson.** Prefer a guarantee the architecture enforces to one a policy promises. Policies are edited by people in a hurry.

### Scope and the authorization contract

*What a credential carries, who may widen it, and what the platform refuses to decide.*

#### ADR-12 · The service reports what a credential carries and never what it permits

**Status:** Accepted  ·  **Shown on views:** 01, 09, 12

*Does the credential service make authorization decisions, or only supply the inputs to them?*

**Context.** The tempting scope expansion is obvious: the service already knows the tenant, the principal and the scope set, so it could answer 'may this request proceed'. Doing so would put every product's authorization rules inside a component that must never be slow and must never be wrong, and would make every product change a change to the credential service. It also collapses two very different failure modes — 'this credential is not valid' and 'this credential may not do this' — into one answer, which is the wrong thing to have happen during an incident.

**Decision.** Verification returns a normalised authorization context — tenant, principal, scope set, constraints, credential class, expiry, staleness — and nothing more. The resource owner decides what that permits. The service publishes the scope vocabulary as machine-readable metadata so consoles, SDKs and gateways agree on what exists, and stops there.

**How it is realised on Google Cloud.** The context is returned as a single structure the caller passes downstream without re-querying, carried as a signed header between the sidecar and the application. Product authorization is a separate concern in each service, reading that context. View 01 draws product authorization explicitly outside the boundary so the omission is a decision rather than an oversight.

| Option | Verdict | Reasoning |
|---|---|---|
| Context only; the resource owner decides | Chosen | Keeps the hot path small and the blast radius of a product change local to that product. |
| Full policy decision point | Rejected | Every product's rules inside the most latency- and correctness-critical component on the platform, with every product change deploying to it. |
| Context plus a coarse allow/deny on scope membership | Rejected | Sounds harmless and is where a policy engine starts. The second request is always for a resource-level condition. |
| Separate policy engine reading this context | Right elsewhere | Right for an organisation that wants centralised authorization. It should be a different service with a different SLO, consuming this one's output. |

**What it buys**

- The verification path stays a lookup, which is what keeps the p99 budget achievable.
- A product authorization bug cannot take down credential verification for the whole platform.
- The two failure modes stay distinguishable in logs and in incident response.

**What it costs**

- Every product implements its own authorization, with the inconsistency that implies.
- Customers see scopes from this service and permissions from products, and must reconcile the two mentally.
- There is no single place to answer 'what can this key actually do', which is a question customers reasonably ask.

**Choose differently when.** Fold authorization in when the platform is one product with one resource model. Split it out again the moment there are two, which will be before anyone expects.

**Why it holds up over time.** This boundary is what allows the platform's product surface to change completely over ten years without the credential service changing at all. It is also the boundary most often eroded by a well-meaning feature request, which is why it is drawn rather than described.

> **Lesson.** A component that must never be slow should own as few reasons to change as possible.

#### ADR-13 · A flat, additive, centrally owned scope vocabulary

**Status:** Accepted  ·  **Shown on views:** 06, 07, 11, 17

*What shape is the scope vocabulary, who owns it, and may an entry ever be removed?*

**Context.** Hierarchical scopes are expressive and become impossible to reason about at a glance, which matters because a developer picks scopes in a console in about eight seconds. Federated ownership scales with the organisation and reliably produces three spellings of 'read'. And removal is the quiet catastrophe: a scope withdrawn from the vocabulary breaks every customer integration holding a credential that carries it, with no warning and no migration path, at a time chosen by an internal team.

**Decision.** Scopes are flat `resource:action` strings, versioned, additive only, and centrally owned by one registry with one reviewer. Removals are rejected at the pipeline gate; a scope may be marked deprecated and stop being offered to new credentials, but it never stops resolving for credentials that already hold it. Each scope carries a grade — read, write or admin — which the staleness rule in ADR-07 consumes.

**How it is realised on Google Cloud.** The vocabulary is a declarative file, linted for synonyms and removals, built into an immutable signed artefact and rolled out in rings (ADR-20). A replay simulation runs the previous day's verifications against the candidate vocabulary and fails the build if any outcome changes. The registry is a service rather than a file at runtime, because the no-escalation check in ADR-14 needs the principal's own scopes at issuance time.

| Option | Verdict | Reasoning |
|---|---|---|
| Flat, additive, centrally owned | Chosen | Legible in a console, safe for existing integrations, and coherent across product surfaces — at the cost of being a bottleneck. |
| Hierarchical scopes with wildcards | Rejected | Expressive, and nobody can tell what payments:* will mean after the next product launch. A wildcard is a scope that grows without review. |
| Federated per-service vocabularies | Deferred | Scales with the organisation, and this is Core Architecture Question 7. The trigger to revisit is the rate of new-scope requests, not a complaint about review latency. |
| Free-form scope strings | Rejected | Maximum flexibility, zero ability to present a coherent consent screen or to grade a scope's consequence. |

**What it buys**

- A five-year-old customer integration keeps working, because nothing is ever removed.
- One vocabulary means a console, an SDK and a gateway describe the same thing the same way.
- Grading is available to the staleness rule without a separately maintained list.

**What it costs**

- The registry owner is on the critical path of every product team's roadmap, which is a real organisational cost and the reason ADR-13 has an open successor question.
- Additive-only means the vocabulary only grows; after ten years it carries scopes nobody remembers the purpose of.
- Flatness means some genuinely hierarchical resources are expressed clumsily.

**Choose differently when.** Federate when the number of product surfaces makes central review the dominant delay, and accept the drift — or invest in cross-vocabulary linting, which is the same central cost wearing different clothes.

**Why it holds up over time.** Additive-only is what makes the vocabulary safe to inherit. A future team can add, deprecate and grade without ever being able to break an existing customer, which is the property that lets the vocabulary outlive its authors.

> **Lesson.** A public vocabulary is an API. Removing an entry is a breaking change, whatever the internal documentation says.

#### ADR-14 · No principal may issue a credential wider than itself, checked at issuance

**Status:** Accepted  ·  **Shown on views:** 03, 13, 21

*What stops key creation from being a privilege-escalation path?*

**Context.** If a principal with read access can create a credential carrying write scopes, then the read/write distinction is decorative: any compromised read-only session becomes a write-capable credential that outlives the session by a year. The check has to happen where the principal's own scopes are known, which is at issuance in the credential service — not in the console, which is one of several clients and can be bypassed by calling the management API directly.

**Decision.** Issuance refuses any scope the creating principal does not itself hold, evaluated in the issuance service against the principal's scopes as asserted by the platform IdP. Scope escalation on an existing credential requires step-up authentication and is recorded with its evidence in audit. The same rule applies to machine principals: a workload cannot mint a credential broader than the workload identity it runs as.

**How it is realised on Google Cloud.** The issuance service resolves the principal from the IdP assertion or the workload identity, queries the scope registry for the principal's held scopes, and computes the intersection. A request for anything outside it fails with an explicit error naming the missing scope, which is safe because the caller is authenticated. Step-up is required for escalation, constraint removal and bulk revocation, and the step-up evidence is written alongside the issuance record.

| Option | Verdict | Reasoning |
|---|---|---|
| Refuse at issuance against the principal's own scopes | Chosen | Enforced at the one place that cannot be bypassed, and cheap because issuance is rare. |
| Check in the console only | Rejected | Bypassed by calling the management API directly, which is documented and encouraged. A control in one client is not a control. |
| Allow escalation with an approval workflow | Deferred | Reasonable for an enterprise tenant with separation of duties, and it is a tenant-configurable extension rather than a change to the default. |
| Allow any scope, detect escalation afterwards | Rejected | Detection after issuance means the credential existed and may already be in a config file somewhere. |

**What it buys**

- A compromised low-privilege session cannot be laundered into a long-lived high-privilege credential.
- The rule is stated once, in one service, and applies to every client of the management API.
- Audit can answer 'who authorised this scope' years later, with the step-up evidence attached.

**What it costs**

- Issuance depends on the IdP and the scope registry being reachable, which is part of why the control plane's availability target is lower than verification's.
- A legitimate delegation — a CI system minting credentials for scopes it does not itself use — must be modelled as a principal holding those scopes, which is more setup than a wildcard would be.

**Choose differently when.** Relax only where every principal is already fully privileged, which usually means the tenant model is too coarse and that is the thing to fix instead.

**Why it holds up over time.** The rule is independent of how identity is represented; whatever replaces OIDC still yields a principal with a scope set. What ages is the enforcement point, and putting it in issuance rather than the console is what makes it survive the arrival of the fifth API client.

> **Lesson.** Enforce a control where it cannot be routed around, not where it is easiest to write.

#### ADR-15 · last_used_at is an observation, never an authorization input

**Status:** Accepted  ·  **Shown on views:** 03, 11, 13, 19

*Should a credential that has not been used for ninety days expire automatically?*

**Context.** Automatic retirement of idle credentials is genuinely valuable: it is the mechanism that keeps credential sprawl from becoming permanent. But making idleness an authorization input changes the nature of the hot path. Usage would have to be recorded durably, at 250,000 writes per second, before a verification could be trusted — and a lossy counter that can revoke something is a mechanism that will eventually revoke a credential that was in fact in use. The value of the feature is real; the cost lands on the one path that must stay a memory read.

**Decision.** last_used_at and call counts are observations: written asynchronously, aggregated in a write-optimised store, accurate to within 5 minutes, and explicitly lossy. Nothing reads them to make an authorization decision. Idle credentials are surfaced to their owners — in the console, in ownership review, and in rotation's report of a predecessor's last use — and a human decides. Automatic expiry on idleness is deferred to Phase 3 and, if built, must not read this counter.

**How it is realised on Google Cloud.** The sidecar emits a sampled usage event off the request path; aggregation writes to Bigtable on a rolling window. The console shows last-used timestamp and region against every credential, labelled as approximate. Rotation's report of the predecessor's last use carries the same label, and the automatic revocation at day 7 is driven by the clock, never by the counter.

| Option | Verdict | Reasoning |
|---|---|---|
| Lossy observation, human decides | Chosen | Keeps the hot path stateless and gives the owner what they actually need: evidence to retire a key with confidence. |
| Durable usage write, automatic expiry on idleness | Rejected | Makes verification a durable write at full request rate, and lets a dropped event revoke a credential that was in use. |
| Periodic sampling with a wide idleness threshold | Deferred | A 180-day threshold makes the loss rate tolerable, which is the shape any future automatic retirement should take — with its own record. |
| No usage tracking at all | Rejected | Cheapest, and it removes the evidence that lets anyone retire anything. Sprawl then becomes permanent. |

**What it buys**

- The verification path stays a read, with telemetry that may be dropped but may never delay an answer.
- Observability cost scales with traffic rather than with correctness requirements.
- The credential owner gets the one field that makes retirement a decision rather than a gamble.

**What it costs**

- Credential sprawl is bounded by quota and by human review, not by automatic retirement — slower and less complete.
- Approximate timestamps invite exactly the misuse this record forbids, so the labelling has to survive every future UI change.

**Choose differently when.** Durable usage tracking is right when verification volume is low enough that a write per request is affordable, or when regulation requires provable last-use rather than observed last-use.

**Why it holds up over time.** The separation between observation and input is the durable part. If automatic retirement is ever built, this record tells a future team exactly what it must not reuse, which is more useful than a prohibition with no reasoning attached.

> **Lesson.** Decide what a number is allowed to cause before deciding how accurate it needs to be.

### Detection and response

*Finding a leaked credential, and what the platform is allowed to do about it on its own.*

#### ADR-16 · Greppable prefixes and a partner endpoint that returns one bit

**Status:** Accepted  ·  **Shown on views:** 01, 05, 09, 15, 20

*How does a repository host tell the platform it has found a live credential, without the platform disclosing anything to an unauthenticated caller?*

**Context.** Public-repository scanning only works if the credential is recognisable. That means publishing a stable, distinctive prefix — which is an intentional disclosure, and the right one: the alternative is a random string no scanner can find. The harder half is the confirmation step. A scanner that finds a candidate needs to know whether it is live, which means calling the platform. That caller cannot be authenticated in any meaningful per-customer way, and whatever it can ask, an attacker with a partial key can also ask.

**Decision.** Credential prefixes are stable, documented and distinctive per class and environment. The partner confirmation endpoint is unauthenticated by design, hard-quota'd per partner, and structurally incapable of returning anything but live-or-not: it never loads the tenant, the scopes or the owner into scope. Own-corpus scanning — internal repositories, logs, support attachments — is in the must-have tier alongside the partner integration.

**How it is realised on Google Cloud.** Cloud Armor in front of the endpoint with a per-partner quota of 20 requests per second, well below any useful search rate against 160 bits of entropy. The handler computes the digest, checks membership and returns a boolean; the code path has no access to the credential record's other columns. Sustained NOT_FOUND rates from a single source raise an alert regardless of quota compliance.

| Option | Verdict | Reasoning |
|---|---|---|
| Unauthenticated one-bit endpoint with a hard quota | Chosen | Partners can act immediately, and the worst an abuser gains is confirmation of a key they already hold. |
| Authenticated partner integration | Rejected | Cleaner on paper. Partners scan for hundreds of vendors and will not hold per-vendor credentials, so the integration does not happen. |
| Platform pulls from partner feeds instead | Deferred | Removes the endpoint entirely and adds latency plus a dependency on each partner's feed format. Worth it only if the endpoint proves abusable. |
| No prefix — random credential strings | Rejected | Discloses nothing and makes the platform undetectable to every scanner, which trades a theoretical disclosure for a real detection capability. |

**What it buys**

- Leaked credentials are found by parties the platform does not have to build or operate.
- The prefix makes a credential legible in a log, an error and a support ticket without exposing the secret.
- The endpoint's blast radius is one bit, enforced by what the code path can reach rather than by a filter on the response.

**What it costs**

- The prefix tells an attacker what they have found, which marginally increases the value of a leak they already hold.
- An unauthenticated endpoint on the public internet is permanent attack surface that must be watched.
- Partner coverage is uneven: keys leaked to private repositories, logs and tickets are invisible to it, which is why own-corpus scanning is not optional.

**Choose differently when.** Pull from partner feeds instead when the endpoint shows abuse that quotas cannot contain, or when a regulator objects to an unauthenticated interface regardless of what it returns.

**Why it holds up over time.** The prefix is the most permanent artefact this platform produces: it appears in customer configuration everywhere and can never be changed without breaking scanners. The one-bit constraint is what keeps that permanence safe.

> **Lesson.** Make the interface incapable of over-disclosing rather than careful not to. Care does not survive a refactor.

#### ADR-17 · Auto-revoke on a confirmed leak; anomaly-only response is opt-in and reversible

**Status:** Accepted  ·  **Shown on views:** 05, 07, 15, 22

*What is the platform allowed to do on its own when it believes a credential is compromised?*

**Context.** A confirmed partner match is near-certain evidence: the exact secret was found in a public commit. Acting automatically there is straightforwardly right — waiting for a human costs minutes during which the credential is public. An anomaly signal is different in kind. First use from a new country is a deployment to a new region. A tenfold rate change is a successful product launch. An unexercised scope is a new feature. Revoking on those takes down a live customer integration for a reason the customer will find absurd, and the trust cost of one such event exceeds the leak it prevented.

**Decision.** On a confirmed leak, the configured response executes automatically within the leak-response SLO: notify, and either revoke, downgrade to read-only scopes, or quarantine pending owner confirmation, per tenant policy and credential class. Automated action on anomaly signals alone is opt-in per tenant. Every automated action is reversible and carries its full decision trail, including what triggered it.

**How it is realised on Google Cloud.** The leak-response engine reads a per-tenant policy keyed by credential class. Downgrade is implemented as a scope-set change on a new credential version rather than a revocation, so it is instantly reversible. The trigger trail — source, evidence, policy version, timestamp — is written to the incident record, and the lifecycle webhook carries the same identifiers so the customer's own systems can act.

| Option | Verdict | Reasoning |
|---|---|---|
| Auto on confirmed, opt-in on anomaly | Chosen | Matches the action's aggressiveness to the evidence's certainty, and puts the false-positive decision with the party that bears it. |
| Auto-revoke on any signal | Rejected | Fastest containment and the fastest route to a customer outage caused by a heuristic. One such event costs more trust than several slow containments. |
| Human confirmation for everything | Rejected | Safe against false positives and adds minutes during which a public credential is live. The 60-second SLO is not achievable with a human in the loop. |
| Downgrade to read-only as the universal default | Deferred | An attractive middle for platforms where read access is not itself sensitive. Not here, where a read scope can reach customer financial data. |

**What it buys**

- Confirmed leaks are contained in under a minute without waking anyone.
- The tenant owns the risk appetite for uncertain signals, which is where the consequence lands.
- A disputed automated action has an answer, because the trigger trail is retained.

**What it costs**

- Anomaly detection is off for most tenants, so the signals it produces are advisory and slower to act on.
- Per-tenant policy is configuration that must be correct, discoverable and defaulted sensibly, or it becomes a support burden.
- Downgrade-as-a-version-change is a second containment mechanism with its own semantics to test.

**Choose differently when.** Auto-act on anomalies when the platform's credentials authorise only low-consequence actions, or when the customer base is internal and an incorrect revocation is a five-minute inconvenience rather than an outage.

**Why it holds up over time.** Detection techniques will improve and the confidence threshold will move. What should not move is the principle that the aggressiveness of an automated action is tied to the certainty of its evidence, and that the party bearing the false-positive cost chooses the setting.

> **Lesson.** Automate in proportion to the evidence, and let whoever pays for a false positive decide the threshold.

### Platform, deployment and operations

*Where it runs, how a change reaches it, and how its central claim is kept honest.*

#### ADR-18 · Verification is active-active everywhere; the control plane has one leader

**Status:** Accepted  ·  **Shown on views:** 07, 08, 16, 22

*What is the topology, and does every component need the same recovery objective?*

**Context.** Treating the whole service as one thing produces either an over-engineered control plane or an under-engineered data plane. The two halves have genuinely different requirements: verification failing for one second is visible to every customer of the platform, while issuance failing for fifteen minutes is an inconvenience a developer works around. Building both to the same standard means paying for the harder one twice, or — more commonly — settling for the easier one everywhere.

**Decision.** Verification is active-active in all three regions with RTO 0: losing a region is a capacity event, not an availability event. The control plane runs a leader region with warm standbys at RTO ≤ 15 minutes. Availability targets differ accordingly: ≥ 99.99% for verification, ≥ 99.9% for the management API. The two are deployed, released and alerted on separately.

**How it is realised on Google Cloud.** GKE in europe-west1, us-east4 and asia-southeast1, each running the product fleet with its sidecars and a regional Memorystore and snapshot bucket. Control-plane workloads run in all three but accept writes only in the leader. Spanner is multi-region rather than per-region because the revocation log's RPO 0 requires a quorum that spans regions; a single-region configuration would make the most important guarantee in the architecture regional.

| Option | Verdict | Reasoning |
|---|---|---|
| Active-active data plane, single-leader control plane | Chosen | Pays for the hard guarantee where it is needed and takes the cheaper one where it is survivable. |
| Active-active everywhere | Rejected | Multi-master issuance means conflict resolution on credential creation and quota, for a write path measured in hundreds per second. Cost with no benefit. |
| Single region for everything | Rejected | Makes regional loss a total outage of verification, which is the one thing that must never stop. |
| Regional Spanner with async cross-region replication | Rejected | Materially cheaper, and revocation durability becomes regional. That is the guarantee the multi-region cost exists to buy. |

**What it buys**

- Regional loss costs capacity, not availability, for the path every request depends on.
- The control plane can be simpler, and its failover exercised without touching verification.
- Two clearly different SLOs make it obvious which alerts wake someone at 3 a.m.

**What it costs**

- Multi-region Spanner is the single largest line in the architecture, justified by exactly one requirement.
- Two deployment topologies and two failover procedures to maintain and rehearse.
- Control-plane failover, being rare, is the procedure most likely to have rotted when it is needed.

**Choose differently when.** Single region is right for a platform serving one geography with a regional availability target. Active-active issuance becomes right if credential creation ever becomes a high-rate machine operation rather than a human one.

**Why it holds up over time.** The split between a plane that must never stop and a plane that may pause is the durable structure. Regions, services and even clouds can be swapped underneath it; what would break it is letting an issuance-path dependency creep into verification, which view 20 draws as a refused crossing for that reason.

> **Lesson.** Give each plane the recovery objective its failure actually deserves, and say out loud which requirement is paying for the expensive one.

#### ADR-19 · A revocation canary, run from a project the platform's operators cannot deploy to

**Status:** Accepted  ·  **Shown on views:** 02, 14, 18

*How does anyone know that revocation still works today?*

**Context.** Every component in the propagation chain can fail silently. A subscriber that stops applying messages reports healthy. A snapshot builder that writes a stale object reports success. A staleness guard with an inverted comparison passes its unit tests and fails in production in exactly one direction. None of these produce errors, and all of them mean a revoked credential keeps working. Instrumenting the feed does not help: the feed will report that it published, which is true and irrelevant.

**Decision.** A canary continuously issues a synthetic credential, verifies it, revokes it and re-verifies it, end to end, through the same paths a real credential takes. Propagation p99 is the measurement from that loop. The canary runs in a separate Google Cloud project, in a region chosen independently of the leader, and the platform's own operators cannot deploy to it. Its silence is alerted on by a dead-man check outside both.

**How it is realised on Google Cloud.** A Cloud Run job in the assurance project runs the cycle every 60 seconds against each region's verification points, recording the interval between revocation acknowledgement and first denial at each point. A dead-man timer external to both projects fires if the canary itself stops reporting, because a stopped canary and a healthy platform produce the same absence of alerts.

| Option | Verdict | Reasoning |
|---|---|---|
| End-to-end canary, externally hosted, dead-man alerted | Chosen | The only signal that catches a silent propagation failure, and the only one whose own failure is also detected. |
| Feed instrumentation and subscriber lag metrics | Rejected | Necessary and insufficient. A subscriber that receives and fails to apply reports zero lag while honouring revoked credentials. |
| Periodic manual drill | Rejected | Proves it worked on the day of the drill. The failures being guarded against appear between drills and persist. |
| Canary inside the platform's own project | Rejected | Cheaper, and it shares a failure domain with the thing it is judging — including the deploy that broke it. |

**What it buys**

- The propagation SLO is a measurement rather than a claim, and a regression is visible within a minute.
- A change that silently breaks the staleness guard or the overlay is caught by the canary rather than by a customer.
- Independence means an outage of the platform does not take the witness with it.

**What it costs**

- A second project, with its own access model, budget and on-call ownership.
- The canary writes real credentials continuously, which is a small permanent load and a set of synthetic records to exclude from every report.
- Its own reliability now matters, which is what the dead-man check is for and one more thing to maintain.

**Choose differently when.** An in-platform canary is acceptable when there is no separate project available and the alternative is no canary. Never accept feed metrics alone as the assurance for a revocation guarantee.

**Why it holds up over time.** The canary tests the contract rather than the implementation, so it survives replacing the bus, the store, the sidecar and the cloud. It is the single artefact most likely to still be valuable in ten years, and the one that keeps the architecture's central claim honest.

> **Lesson.** If a guarantee cannot fail loudly, buy a witness — and make sure the witness cannot go quiet unnoticed.

#### ADR-20 · Config artefacts are immutable, ring-rolled, and refused rather than fallen back from

**Status:** Accepted  ·  **Shown on views:** 17, 18, 22

*How does a scope vocabulary or policy change reach every verification point without being able to break them all at once?*

**Context.** Configuration is the fastest path to a total outage in this architecture, because a verification point applies it immediately and there are thousands of them. A bad vocabulary or a malformed policy reaches all of them within seconds of publication, and the natural failure behaviour of a config parser — fall back to a default — is catastrophic here, because the permissive default is 'allow'. The same change that would be a slow, reviewable code deploy is, as configuration, an instant fleet-wide event.

**Decision.** Configuration is built into immutable, signed artefacts with provenance, not pushed live from a database. Rollout is staged: a canary ring of three services for thirty minutes with an error-rate watch and automatic rollback, then an internal ring, then the remainder. A verification point that cannot parse an artefact keeps the last good one and reports its version; it never falls back to a permissive default. Config version spread across the fleet is a monitored quantity.

**How it is realised on Google Cloud.** The artefact is built in the same pipeline as the vocabulary lint and the replay simulation (ADR-13), signed, and published to Cloud Storage with its provenance. Verification points pin the last good version locally, so a restart during a bad rollout comes back on the last known-good rather than on the broken artefact. The rollback is a single command that re-pins a version, not a rebuild.

| Option | Verdict | Reasoning |
|---|---|---|
| Immutable artefacts, ring rollout, refuse-and-pin | Chosen | Makes the blast radius of a bad config a ring rather than the fleet, and makes the failure mode 'stale' rather than 'permissive'. |
| Live config from a database | Rejected | Simplest and fastest, with no staging, no rollback and no provenance. The fastest available way to take the platform down. |
| Config shipped in the application binary | Rejected | Safest rollout there is, and a scope addition now waits for a full deployment cycle across 180 services. |
| Parse-fail to a permissive default | Rejected | The conventional behaviour, and here it converts a syntax error into an authorization bypass. |

**What it buys**

- A poison config is contained to a ring and rolled back automatically.
- Restart during a bad rollout is safe, because the last good version is pinned locally.
- Version spread across the fleet makes an incomplete rollout visible instead of assumed.

**What it costs**

- A scope addition takes hours to reach the whole fleet rather than seconds, which is a real friction for product teams.
- Artefact building, signing and provenance is a pipeline to own.
- Pinned last-good versions mean a point that has been isolated for a long time is running old policy, which is a second staleness to reason about.

**Choose differently when.** Live config is acceptable where a bad value degrades one feature rather than the authorization decision. The distinction is whether the config can make the system permissive, not how quickly it needs to change.

**Why it holds up over time.** Immutability and staged rollout are stable practices, and the refuse-rather-than-default rule is the part most likely to be quietly reversed by a future engineer fixing a parsing bug. It is recorded here with its reasoning so that reversal is a decision rather than a patch.

> **Lesson.** Configuration that can make a system permissive deserves the same rollout discipline as code, and the same refusal to guess.

## Every package used, in one table

The terms this package uses in a specific way, and what each one is doing in the architecture.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| Key identifier (kid) | The public, non-secret portion of a credential that names the record. | Lets a log line, an error message and a scanner refer to a credential without holding its secret. | Logging a truncated secret, which is a secret. |
| Prefix | A fixed leading string encoding credential class and environment, for example sk_live_. | Makes a credential greppable by scanning partners and makes a test key failing against live legible in the error. | An opaque random string, which no scanner can recognise. |
| Pepper | A secret key held outside the credential store, mixed into every digest. | Makes a stolen database worthless without a second compromise. | A per-record salt, which is stored beside the record and therefore stolen with it. |
| Projection | A verification point's in-memory copy of active credentials and revocations. | Turns verification into a memory lookup with no network dependency. | A cache, which implies demand-filling and a cold path. |
| Revocation overlay | The additive, monotonic set of revocations applied on top of a projection. | Lets revocations propagate independently of, and faster than, credential state. | Updating the credential record, which couples the two. |
| Staleness | How far behind the revocation feed a projection currently is. | Returned on every response and compared against a declared ceiling; the basis of the fail-closed rule. | A TTL, which expresses an intention rather than a measurement. |
| Staleness ceiling | The bound past which a verification point stops honouring privileged scopes. | Converts an unbounded silent failure into a bounded, visible one. | Serving indefinitely, which is what every cache does by default. |
| Tenant epoch | A monotonic counter per tenant; credentials issued before the current value are invalid. | Makes bulk revocation cost one write and one message regardless of how many credentials it kills. | N individual revocations, whose propagation time scales with the incident. |
| Scope grade | A classification of each scope as read, write or admin, carried in the vocabulary. | Lets the staleness rule degrade by consequence rather than by an out-of-band list. | A hand-maintained list of sensitive scopes, which drifts from the vocabulary. |
| Verification point | Any process that can answer a verification: a library, a sidecar, or the remote API. | The unit that holds a projection, tracks staleness and applies the feed. | "The verification service", which implies a central component this design does not have. |
| Revocation canary | A synthetic credential issued, verified, revoked and re-verified continuously in production. | Makes propagation p99 a measurement rather than a claim in a document. | Instrumenting the feed, which cannot detect a subscriber that silently stopped applying. |
| Live-or-not endpoint | The unauthenticated partner interface that confirms whether a candidate string is a live credential. | Lets a scanning partner act without the platform disclosing tenant, scopes or owner. | A lookup API, which would hand an unauthenticated caller the blast radius. |