Architecture Views

Does a long-lived developer credential carry its own scopes and expiry, verified by a signature, or is it an opaque handle that must be looked up?

Context

A self-contained signed token verifies with no state at all: the verifier holds a public key and reads the claims out of the token. That is the right answer inside a trust boundary for a credential that lives ninety seconds. A developer API key lives eighteen months, sits in a CI variable, and its defining event is the day it appears in a public commit. On that day the platform is asked one question: how long does it stay useful to whoever found it. A self-contained token's honest answer is "until it expires", and the usual remedy — a denylist consulted on every verification — reintroduces exactly the state the token format existed to avoid, with none of the design attention that state would otherwise have received.

Decision

Long-lived credentials are opaque: a public prefix encoding class and environment, a key identifier usable for lookup and logging, and a high-entropy secret. Nothing about what the credential may do travels inside it. Authority is resolved by lookup, always. Short-lived self-contained tokens are permitted in Phase 2 but only as something minted from an opaque credential, which moves the revocation problem to the exchange rather than removing it.

How it is realised on Google Cloud

Format `sk_live_<kid>_<secret>`, with the class and environment in the prefix so a scanner, a log line and an error message can all identify it without the secret. The credential record lives in Cloud Spanner, tenant-partitioned. Nothing signs anything: there is no key pair, no JWKS endpoint, and no verifier that could honour a credential on the strength of the credential alone.

Options weighed

• ChosenOpaque handle with lookup: Revocation is immediate and total, and the credential discloses nothing about the tenant. Costs state at verification time, which ADR-02 and ADR-03 are about.
• RejectedSigned self-contained token (JWT-style): Stateless verification, but valid until expiry whatever anyone does. A denylist to fix that is the lookup you were avoiding, built late and designed less.
• DeferredHybrid: short-lived tokens minted from a long-lived key: Right for machine-to-machine traffic and planned for Phase 2. Does not remove the problem: the long-lived key at the exchange is still this decision.
• Right elsewhereSigned token with a very short expiry and no revocation: Correct inside a service mesh where credentials live for seconds and the issuer is on the same network. Not correct for a string a customer pastes into a config file.

Consequences

What it buys

• Revocation becomes possible at all, which is the whole product.
• The credential leaks nothing: a key in a public commit does not disclose the tenant, the scopes or the environment beyond what the prefix says deliberately.
• Scope changes take effect on the next verification, without waiting for a token to expire.

What it costs

• Every verification needs state, which forces the projection, the feed and the staleness machinery — most of this architecture.
• A verification point that has never been warmed cannot verify anything, where a signature check would have worked from the first request.

Choose differently when

Choose a self-contained token when credentials live minutes rather than months, when the verifier cannot hold state at all, or when there is no requirement to revoke faster than expiry. A webhook signature and a service-mesh identity are both in that category.

Why it holds up over time

This decision depends on no technology. Spanner, Pub/Sub, Google Cloud and HMAC can all be replaced without touching it. What would break it is a future team adding a 'fast path' that reads scopes out of the credential to avoid a lookup — which is why the refused crossing is drawn explicitly in view 20 rather than left as a convention.

Where does verification execute — in the calling process, in a sidecar beside it, or in a central service the caller calls?

Context

ADR-01 made verification stateful. The naive consequence is a central verification service, which is the most common design and the one that fails predictably: at 250,000 requests per second it is a network hop on every request in the platform, it becomes the single most critical dependency anything has, and the teams calling it respond exactly as you would expect — they cache the answer for five minutes with no revocation awareness, or they skip the check under load. A verification that is expensive gets skipped, and a verification that gets skipped is worse than one that never existed, because nobody knows which requests were checked.

Decision

Verification executes locally at the calling service: a library in-process where the language allows, an Envoy `ext_authz` sidecar otherwise. The verification point holds a projection of active credentials and a revocation overlay, and answers from memory. A remote verification API exists for thin clients that genuinely cannot hold state, is bounded at p99 ≤ 15 ms, and is explicitly not the intended path.

How it is realised on Google Cloud

Sidecar container in each GKE pod, exposing `ext_authz` over localhost to the Envoy in front of the application; a Go and a Java library for the services that would rather link it. Both share one projection implementation and one revocation-feed client, so there is one staleness behaviour to reason about and one to test. The remote API runs in each region backed by a Memorystore-held projection, and its call rate is a reported metric because a rise in it means a team could not adopt the library.

Options weighed

• ChosenLocal library plus sidecar: No network hop, no shared dependency, and nobody is tempted to cache the answer because the answer is already local.
• RejectedCentral verification service: One hop per request platform-wide, and the most critical dependency in the estate. Reliably produces badly-cached client-side workarounds that outlive the incident that caused them.
• RejectedGateway-only verification: Cheap, but only the edge is checked. Internal replay, async workers and service-to-service calls then trust an upstream assertion nobody re-verifies.
• Right elsewhereVerification in the data store's own authorization layer: Elegant when every access goes through one store. Not available to a platform with 180 services and several storage technologies.

Consequences

What it buys

• Verification costs a memory lookup, which makes 'verify on every request' a decision nobody has to defend in a budget review.
• No shared runtime dependency: a control-plane outage does not stop a single request being served.
• One implementation of the staleness rule, rather than one per calling team.

What it costs

• Memory in every pod, proportional to the number of active credentials — the assumption ADR-03 is about.
• A library is a product: versions, upgrades, a deprecation path and a support burden the platform team now owns.
• The remote API remains as an escape hatch and will be used by someone; it needs its own capacity and its own SLO.

Choose differently when

Choose a central service when the calling fleet is small and homogeneous, when credentials number in the thousands rather than the millions, or when the per-request latency budget is generous enough that a hop is invisible. Choose gateway-only when the platform genuinely has one ingress and no internal re-entry.

Why it holds up over time

The sidecar mechanism will age — service meshes come and go — but the property that matters is that verification is co-located with the caller. Whatever replaces Envoy in ten years will still be a place to put this, and the contract (a decision plus a staleness figure) is unchanged by the move.

Does a verification point hold every active credential, or only the ones it has seen?

Context

A demand-filled cache is smaller and obviously correct in steady state. Its problem is the cold path: on a miss it must read the durable store. That path runs precisely when it is most dangerous — a fleet restart, a deployment, a region rejoining after a partition — and turns a rollout into a load spike against the one store that must not be slow. It also makes the first request for each credential slower than the rest, which is a timing signal about which keys are in use. A full projection removes the cold path entirely, at the cost of memory proportional to the whole platform at every point.

Decision

Each verification point holds a projection of all active credentials for the tenants it serves, warmed from a regional snapshot at start-up, with admission gated on readiness. There is no read from the credential store on the request path, and such a read is counted and alerted on as an exception rather than budgeted as a cost.

How it is realised on Google Cloud

The snapshot builder writes a regional object to Cloud Storage every 60 seconds, holding key identifier, digest, pepper version, scope set, constraints and expiry for every active credential. A starting verification point loads the snapshot, then replays the revocation feed forward from the snapshot's sequence number before it reports ready. At the assumed 2.5 million credentials and roughly 160 bytes per entry this is about 400 MB per point, which fits the platform's standard pod but not its smallest.

Options weighed

• ChosenFull projection, snapshot-warmed: No cold path at any point in the lifecycle, and constant-cost answers for hits and misses alike.
• RejectedDemand-filled cache with a TTL: Small, but the cold path exists exactly when the store is least able to serve it, and the TTL is a second, undeclared staleness mechanism.
• DeferredPartitioned projection by key-prefix shard: Keeps the no-cold-path property with a fraction of the memory, at the cost of routing. This is the fallback if the memory measurement in the prototype fails.
• RejectedBloom filter of valid identifiers plus a store read on a hit: Cheap negative answers, but every legitimate request still reads the store — the cold path made permanent rather than removed.

Consequences

What it buys

• A restart storm is a snapshot-download event rather than a thundering herd against Spanner.
• Hits and misses cost the same, so response timing discloses nothing about which identifiers exist.
• A partitioned region keeps verifying from what it has, with its staleness stated.

What it costs

• About 400 MB of memory in every pod running a sidecar — a real cost, multiplied by the fleet.
• The snapshot path becomes a dependency of start-up, and a stale or corrupt snapshot is a new failure mode.
• Credential growth is now a capacity planning input for every team on the platform, not just for this one.

Choose differently when

Choose a demand-filled cache when credentials number in the tens of thousands, when the working set is a small fraction of the whole, or when verification points are numerous and small. Choose the partitioned projection as soon as memory per point exceeds what the smallest deployable unit can carry.

Why it holds up over time

The number that decides this — active credentials times bytes per entry, against pod memory — moves in both directions over a decade. The decision is therefore written with its trigger attached, and the partitioned fallback is designed rather than hypothetical, so crossing the threshold is a planned migration rather than an incident.

What does verification return when a credential is unknown, revoked, expired or constrained out — and how much does the answer disclose?

Context

The obvious implementation returns quickly for an unknown identifier and more slowly for a known one, because the second does more work. That difference is an oracle: an attacker with a partial key can learn which identifiers exist by timing. The same applies to error text — 'unknown key' versus 'key revoked' distinguishes a guess from a hit. Meanwhile the operator needs to tell those cases apart to answer a support ticket, so the information cannot simply be discarded.

Decision

Verification returns VALID with an authorization context, or INVALID with one of a closed set: NOT_FOUND, REVOKED, EXPIRED, SUSPENDED, CONSTRAINT_FAILED, TENANT_DISABLED. The reason is recorded in telemetry and available to the operator and to the owning tenant; it is not returned to an unauthenticated caller, which receives a single undifferentiated failure. The work done on the invalid path is padded to the shape of the valid path so the two do not differ in timing.

How it is realised on Google Cloud

The sidecar computes the digest and performs the overlay check regardless of whether the identifier resolves, discarding the result on a miss. The API response carries an opaque failure and a request identifier; the reason is emitted at 100% sampling to the telemetry path and surfaced in the console against the credential when the caller can be attributed to a tenant.

Options weighed

• ChosenClosed reason set, disclosed only to the tenant: The operator and the owner can debug; an outsider learns one bit. The padding closes the timing channel that the reason set would otherwise reopen.
• RejectedDetailed reasons returned to the caller: Excellent developer experience and a working enumeration oracle. The developer experience is recovered through the console, where the caller is authenticated.
• RejectedA single opaque failure, recorded nowhere: Discloses nothing and makes every support conversation guesswork, which pushes people toward wider keys and longer expiries.
• Right elsewhereRate-limit the caller instead of padding: Sufficient where the attacker cannot distribute the guessing. Here both are applied, but padding is what holds when the source is a botnet.

Consequences

What it buys

• Response timing and content disclose nothing about which key identifiers exist.
• Support and self-service debugging keep the full reason, through an authenticated path.
• Denial reasons are complete in telemetry, which is what makes the anomaly signals in ADR-17 possible.

What it costs

• The invalid path does work it does not need to do, which costs CPU on exactly the requests you would rather drop cheaply.
• Padding is a property that must be tested, and it will regress silently the first time someone optimises the miss path.

Choose differently when

Return detailed reasons directly when the caller is always authenticated by a second factor before reaching verification, or when the credential space is small enough that enumeration is a non-threat.

Why it holds up over time

Timing side channels do not go away with better hardware; if anything, more predictable execution makes them easier to exploit. The rule — equal work on both paths — is stable, and the test for it is a benchmark that can live in CI for a decade.

Is a revocation a status column on the credential record, or a separate structure with its own durability guarantee?

Context

Revoking by updating a status column is simpler and wrong in one specific way: the revocation now inherits the credential store's replication behaviour and its recovery objective. A restore from a backup taken before the revocation silently un-revokes the key. A cross-region replication lag delays the revocation exactly as it delays everything else. And a bulk revocation becomes a hundred thousand row updates propagated as a hundred thousand changes. The asymmetry is the point: losing an issuance is recoverable, because the customer can issue another. Losing a revocation is not recoverable, because the credential is already in a public commit.

Decision

Revocation state is an independent append-only log with a monotonic sequence, replicated and evaluated separately from credential records, carrying RPO 0. A revocation is acknowledged to the caller only after a durable multi-region quorum write. Once acknowledged it can never be undone — not by replication catch-up, not by cache repopulation, not by a restore from backup.

How it is realised on Google Cloud

A Spanner table in the multi-region `nam-eur-asia1` configuration, written with a commit that spans regions before the API returns, and read by the feed publisher through a change stream. Restores of the credential-record tables are always accompanied by a replay of the revocation log forward from the restore point, enforced in the restore runbook and exercised in the prototype scenarios.

Options weighed

• ChosenIndependent append-only log, quorum-acknowledged: Revocation gets its own durability, its own ordering and its own propagation, none of which are hostage to the credential store's behaviour.
• RejectedStatus column on the credential record: Simple, and a restore un-revokes keys. The failure is silent and arrives during an incident, which is the worst combination available.
• RejectedTombstone rows with a TTL: Bounded storage, and the revocation expires. A credential whose revocation has aged out is live again, which is precisely the class of bug this service exists to prevent.
• Right elsewhereDistributed consensus group per region: Right when there is no managed multi-region store available. Here it is operational burden the platform can buy its way out of.

Consequences

What it buys

• An accepted revocation survives the loss of a region and the restore of any store.
• Revocation ordering is independent of credential replication, so propagation latency is a property this team controls.
• Bulk revocation has somewhere to be cheap — see ADR-08.

What it costs

• A multi-region quorum write puts roughly 150 ms into the revoke API call, which is invisible to a human and would matter if revocation were a high-rate operation. It is not.
• A second store to operate, back up, monitor and reason about.
• Every restore procedure in the platform now has an extra mandatory step, which is exactly the kind of step that gets skipped unless it is tested.

Choose differently when

Use a status column when revocation is rare, non-urgent and the store is single-region anyway — an internal tool with ten users. The moment revocation is an incident-response action, it needs its own durability.

Why it holds up over time

The claim 'an accepted revocation is never lost' is a contract, not an implementation. Spanner can be replaced by whatever offers a multi-region quorum commit in ten years, and the restore-plus-replay rule transfers unchanged. What keeps it honest is the prototype scenario that restores a pre-revocation backup and checks the key is still dead.

How does a revocation reach thousands of independent verification points, and how is the mechanism's own failure detected?

Context

Three mechanisms are available and each fails differently. A push bus is fast and fails silently: a subscriber that stops receiving looks identical to a quiet period. A pull feed is self-healing — a point that misses a poll simply catches up — but adds a floor to propagation equal to the poll interval. A probabilistic structure such as a cuckoo filter of revoked identifiers is small enough to distribute anywhere, and its false positives here mean denying a valid credential, which is a customer outage caused by a data structure's sizing.

Decision

Revocations are pushed over Pub/Sub for latency, with a 5-second poll fallback for self-healing, and a heartbeat on the same channel so that silence is distinguishable from quiet. A verification point tracks the highest sequence number it has applied; falling behind is a measured quantity, not an unknown. The probabilistic filter is rejected for the primary path and reconsidered only above roughly ten thousand verification points.

How it is realised on Google Cloud

A global Pub/Sub topic carrying ordered revocation records, each with its sequence number; subscribers apply them additively to the overlay. A separate poll endpoint returns everything after a given sequence, called every 5 seconds regardless of push traffic. The heartbeat is a record on the same topic emitted every 10 seconds whether or not there are revocations, so its absence is unambiguous.

Options weighed

• ChosenPush plus poll plus heartbeat: Push latency, poll self-healing, and the one thing neither gives alone — a positive signal that the channel is alive.
• RejectedPush only: Fastest, and its failure mode is a verification point that honours a revoked credential forever while reporting perfect health.
• RejectedPoll only: Simplest and most robust, but the poll interval becomes the propagation floor. At the 2-second p50 target the polling rate would itself be a load problem.
• DeferredCuckoo filter distributed to every point: Tiny and attractive at very large fleets. A false positive denies a valid credential, so it needs a confirmation path — which is a store read, and the cold path returns.

Consequences

What it buys

• Propagation p50 tracks push latency while recovery from a missed message needs no intervention.
• A dead channel is detected in seconds by every point independently, which is what makes the staleness rule in ADR-07 enforceable.
• Sequence numbers make 'how far behind is this point' a number rather than a guess.

What it costs

• Two delivery paths to build, test and keep consistent, plus the heartbeat as a third thing that can break.
• Poll traffic at 5-second intervals from every verification point is a constant baseline load the feed must serve even when nothing is happening.

Choose differently when

Poll-only is right when the propagation target is measured in minutes rather than seconds. The filter becomes right when the number of verification points makes per-point state untenable and a confirmation round trip is affordable on the small fraction of requests it affects.

Why it holds up over time

Message buses will change. What survives is the three-part structure: a fast path, a self-healing path, and a positive liveness signal. A design that has all three can swap any one of them; a design that has only the first cannot tell when it has stopped working.

What does a verification point do when it can no longer prove that its projection is fresh?

Context

This is the decision that determines whether the architecture is honest. A point that serves from an old projection indefinitely will eventually honour a revoked credential, and will do so while reporting success. Failing closed protects against that and takes a customer's integration down during an outage the customer did not cause — the platform converting its own incident into theirs. Failing open does the reverse and makes the revocation SLO a fiction the moment the feed breaks. Neither pure answer is defensible for a platform whose credentials authorise both reading an invoice and issuing a refund.

Decision

Every verification response carries the projection's staleness. Past 120 seconds without a successful feed poll, a verification point refuses credentials for write- and admin-graded scopes while continuing to serve read-graded scopes, according to per-tenant policy. The grading comes from the scope vocabulary, so the behaviour is derived from the scope's declared grade rather than from a list maintained separately.

How it is realised on Google Cloud

The staleness guard runs inside the sidecar, comparing wall-clock time against the last successful poll and the last applied sequence number. The scope grade travels in the snapshot and the feed, so the guard needs no additional lookup. The value is returned in the authorization context and exported as a histogram, so the fleet-wide staleness distribution is visible rather than inferred.

Options weighed

• ChosenScope-graded fail-closed at a declared ceiling: Refuses what a stale projection could actually get wrong in a costly way, while an outage does not stop customers reading their own data.
• RejectedFail closed entirely: Simplest to reason about and to test. A feed outage becomes a total platform outage, which is a price the business will refuse after the first occurrence — and then fail open badly.
• RejectedFail open past the ceiling: Available, and makes the revocation SLO untrue exactly when it is being relied on. The failure is invisible, which is what disqualifies it.
• RejectedNo ceiling — serve whatever is held: The default behaviour of every cache, and the reason a revocation becomes a press release.

Consequences

What it buys

• Staleness is a number in the response rather than an unstated property, so callers can make their own decisions with it.
• The worst outcome of a propagation failure is bounded and known in advance rather than discovered.
• Read traffic — the majority — survives a feed outage.

What it costs

• Two behaviours to specify, implement and test per scope grade, and a grading that must be right in the vocabulary.
• A feed outage still refuses writes platform-wide, which will be an incident with customer impact and a difficult conversation.
• The 120-second ceiling is an invented number and the one most likely to be argued down under pressure.

Choose differently when

Fail closed entirely when every scope is consequential — a payments-only platform. Fail open when credentials authorise nothing that cannot be undone and availability is the dominant concern, and say so explicitly rather than arriving there by omission.

Why it holds up over time

The ceiling's value will change with measured propagation data; the structure will not. Because staleness is on the response, any future propagation mechanism can be evaluated against the same contract, and a regression shows up as a distribution shift rather than as an incident.

What happens when a customer needs every credential issued before a certain moment to stop working at once?

Context

This is the request after a laptop is stolen, a contractor leaves, or a repository is found to have been public for a year. Done naively it is a hundred thousand individual revocations: a hundred thousand log records, a hundred thousand feed messages, and a propagation time that scales with the size of the incident — slowest exactly when it matters most. It also makes the feed's worst-case load a function of an event nobody controls.

Decision

Each tenant carries a monotonic revocation epoch. A bulk revocation increments it, in one ordered write, and every credential issued before that epoch is invalid from that moment. Propagation cost is one message regardless of whether the bump kills one credential or a million. Individual revocations remain per-credential records; the epoch handles the set case.

How it is realised on Google Cloud

`tenant.revocation_epoch` in Spanner, written in the same transaction that appends the epoch record to the revocation log. Each credential record carries its issuance epoch; the verification check is a single integer comparison alongside the overlay lookup, adding no measurable cost to the hot path. Bulk operations scoped more narrowly than a whole tenant — by principal, by credential class, by creation window — are expressed as a filtered epoch, evaluated the same way.

Options weighed

• ChosenTenant epoch with a comparison at verification: Constant propagation cost, constant verification cost, and the incident's size stops being the feed's problem.
• RejectedN individual revocation records: Uniform and simple, with propagation time proportional to incident size. The worst case is unbounded and arrives during a security incident.
• RejectedDisable the tenant entirely: Achieves containment by taking the customer offline, which is a different product decision wearing a security costume.
• Right elsewhereBatch revocations into chunked messages: Reasonable when there is no per-credential epoch to compare against. Reduces the constant but keeps propagation proportional to the set.

Consequences

What it buys

• 100,000 credentials revoked and propagated within the same window as one — the requirement's bulk target is met by construction rather than by capacity.
• The feed's worst-case load is independent of incident size.
• Narrower bulk operations reuse one mechanism and one code path.

What it costs

• Every credential record must carry its issuance epoch, and every verification must compare it — a small permanent cost on the hot path.
• An epoch bump is blunt: a credential issued a second before the cut dies with the rest, and the customer must re-issue it.
• Reasoning about 'why did this key stop working' now has two mechanisms to check rather than one.

Choose differently when

Skip the epoch when bulk revocation is genuinely rare and small — a few hundred credentials at a time — and the propagation target has room for it. The mechanism is cheap but not free, and it complicates the explanation of why a credential failed.

Why it holds up over time

Epochs are a decades-old technique precisely because they are independent of the storage and transport underneath. As the platform grows, the epoch's advantage grows with it: the naive design gets worse with scale and this one does not.

How is a credential secret stored so that a stolen database is worthless, without making verification unaffordable?

Context

The standard answer for storing a secret is a memory-hard key derivation function — Argon2id or scrypt — deliberately slow to make offline cracking expensive. That answer is correct for passwords, which are low-entropy and chosen by humans. It is wrong here for two reasons. A platform-generated secret carries 160 bits of entropy from a CSPRNG, which is not brute-forceable regardless of how fast the digest is; the slowness buys nothing against that input. And verification happens 250,000 times a second on the request path, where a deliberately expensive function is not merely costly but architecture-breaking.

Decision

Credentials are stored as HMAC-SHA-256 of the secret under a pepper held in Cloud HSM, with the pepper version recorded alongside. The security against an offline attack comes from the pepper being outside the database rather than from the digest being slow. Secrets are required to carry at least 160 bits of entropy, which is what makes this sound — the entropy requirement and the digest choice are one decision, not two.

How it is realised on Google Cloud

Cloud KMS with a hardware protection level holds the pepper; issuance calls KMS to compute the digest. Verification points cache the pepper by version in memory, fetched over workload-identity-authenticated calls at start-up and on version change, which is why KMS being unreachable stops issuance but not verification. The pepper is never written to Spanner, to a snapshot, to a backup or to a log.

Options weighed

• ChosenHMAC-SHA-256 with an HSM-held pepper: Microseconds per verification, and a database dump alone yields nothing. Rests entirely on the pepper being genuinely isolated and on the entropy requirement holding.
• RejectedArgon2id or scrypt: The right answer for passwords. Here it defends against an attack the entropy already prevents, at a cost that makes local verification impossible.
• RejectedPlain SHA-256, no pepper: Fast, and a stolen database plus a rainbow table over the key format is a real attack once entropy assumptions slip even slightly.
• DeferredSplit scheme — fast digest plus a slow proof-of-possession tier: Worth revisiting if customer-chosen credential material is ever accepted, which would break the entropy assumption this decision depends on.

Consequences

What it buys

• Verification costs microseconds, which is what makes ADR-02 possible at all.
• Exfiltration of the credential store alone does not permit offline forgery or cracking.
• The pepper's separate custody gives ADR-10 its rotation story.

What it costs

• Security depends on the pepper's isolation. Whatever compromises the database must not also reach KMS, which is a claim about the deployment, not about the cryptography.
• The 160-bit entropy requirement is load-bearing and invisible in the digest code; a future change accepting customer-supplied key material would silently invalidate this decision.
• KMS becomes an issuance-path dependency with its own availability characteristics.

Choose differently when

Use a memory-hard KDF the moment any credential material is chosen by a human or supplied by a customer. Use a plain fast hash only where there is no store to steal.

Why it holds up over time

SHA-256 and HMAC have a long remaining life, and the construction is agile: the pepper version field already carries an algorithm identifier, so moving to a different primitive is the same operation as rotating a pepper. The decision's real dependency is the entropy requirement, which is why it is written into this record rather than left in an issuance function.

What actually happens on the day the credential store is exfiltrated?

Context

ADR-09 makes a stolen database worthless while the pepper stays secret. The question that follows is what the response looks like if the pepper itself is suspected — or simply when a scheduled rotation comes due. Without versioning, changing the pepper invalidates every digest in the store, which means asking every customer to re-issue every credential: an event no platform survives gracefully and therefore an event that never happens, which means the pepper is never rotated, which means the scheduled rotation is a policy nobody has ever executed.

Decision

Every credential record carries the pepper version used to compute its digest. Verification selects the pepper by the record's version, so several versions are valid at once. Rotation issues a new version, digests new credentials under it, and migrates existing records opportunistically — on next successful verification, where the secret is momentarily in hand — with a bounded campaign for the remainder. Annual rotation is scheduled and, critically, exercised.

How it is realised on Google Cloud

Cloud KMS key versions map one-to-one to pepper versions; verification points cache the small set of live versions. On a successful verification under an old version, the sidecar reports the identifier to the control plane, which re-digests the secret under the current version within the same request. Credentials never verified during the campaign are re-digested at next use or expire; the version distribution is a reported metric, so a stalled migration is visible.

Options weighed

• ChosenVersioned pepper with opportunistic migration: Rotation without customer impact, which is what makes it something the team will actually do rather than write down.
• RejectedSingle pepper, rotation by mass re-issuance: Simple and never executed. A rotation procedure nobody can afford to run is equivalent to no rotation procedure.
• DeferredPer-tenant peppers: Smaller blast radius per compromise, and a KMS call pattern and cache footprint that grow with tenant count. Revisit if a tenant ever requires cryptographic isolation by contract.
• RejectedRe-encrypt digests in place with a new pepper: Not possible: an HMAC is not re-keyable without the input, and the input is the secret, which the platform does not have.

Consequences

What it buys

• Store compromise is answered by rotating one key, not by a platform-wide customer event.
• Rotation is routine, so the procedure is exercised and known to work when it is needed urgently.
• Old versions can be retired on a schedule with visible progress rather than by assumption.

What it costs

• Verification points hold several pepper versions, and the migration campaign is a long tail that needs owning.
• A credential never used again keeps its old digest until expiry, so retiring a version fully takes as long as the longest credential lifetime.
• The opportunistic re-digest path touches the secret during verification, which is a code path that must be as carefully reviewed as issuance.

Choose differently when

A single pepper is acceptable when the credential population is small enough that mass re-issuance is a real option, or when credentials are short-lived enough that rotation happens naturally within a month.

Why it holds up over time

Versioning is what turns any cryptographic choice into a reversible one. Its value increases over a decade: the primitive chosen today will be questioned, and the version field is the mechanism by which that question has a cheap answer.

Can the platform ever show a customer their credential secret again?

Context

Every support organisation eventually asks for this. A developer has lost the secret, their deployment is broken, and re-creating the key means touching production configuration. Storing the secret encrypted — recoverable under a KMS key, behind a strong access control — solves that conversation and creates a far worse one: there is now a path, however well guarded, by which the platform can produce a customer's live credential. That path will be used by an insider, exposed by a misconfiguration, or subpoenaed. It also destroys the claim the platform makes to its customers about what it holds.

Decision

The secret is generated, digested and returned inside a single request, and no recoverable copy exists in any store, backup, log, export or support tool. A restore of every store the platform operates cannot revive one secret. The lost-secret problem is answered by rotation — issue a successor, overlap, retire the predecessor — rather than by recovery.

How it is realised on Google Cloud

The secret exists in the issuance service's memory for the duration of one response. Logging middleware redacts anything matching the credential format at the emitter, not at the sink. The console shows the secret once behind an explicit affordance with a one-click copy, and thereafter only the prefix and identifier. Envelope encryption with per-tenant CMEK covers the records that do exist, which are digests.

Options weighed

• ChosenShow once, never store: A structural guarantee rather than a policy one. Costs one difficult moment in the developer journey, recovered by making rotation trivial.
• RejectedStore encrypted, recoverable under strong controls: Solves a real support problem by creating a path to every customer's live credentials. The controls are the only thing standing between an insider and the platform's worst day.
• DeferredEscrow with the customer's own KMS key: Moves custody to the customer, which is defensible. Worth revisiting only if a regulated customer requires escrow by contract, and it should be per-tenant and visible.
• RejectedShow once, with a 5-minute re-display window: A small window is still a stored secret with a TTL, and the TTL is the only thing preventing the attack.

Consequences

What it buys

• The platform can state truthfully that it cannot reproduce a customer's secret, and that statement survives an insider and a backup misconfiguration.
• Backups, exports and support tooling are categorically out of the secret's blast radius.
• There is no recovery endpoint to attack, rate-limit or audit.

What it costs

• A lost secret is a rotation, which touches production configuration. This is the trough in the developer journey and it is deliberate.
• Redaction must be enforced at every emitter, and one careless log line in one service reintroduces the risk this decision removes.

Choose differently when

Escrow becomes right when the customer, not the platform, holds the key, and the customer has asked for it in writing. It is never right as a convenience.

Why it holds up over time

Structural guarantees outlive organisational ones. Ten years and four teams later, nobody can quietly relax this: making secrets recoverable would require changing the issuance path, which is a visible and reviewable act rather than a permission change.

Does the credential service make authorization decisions, or only supply the inputs to them?

Context

The tempting scope expansion is obvious: the service already knows the tenant, the principal and the scope set, so it could answer 'may this request proceed'. Doing so would put every product's authorization rules inside a component that must never be slow and must never be wrong, and would make every product change a change to the credential service. It also collapses two very different failure modes — 'this credential is not valid' and 'this credential may not do this' — into one answer, which is the wrong thing to have happen during an incident.

Decision

Verification returns a normalised authorization context — tenant, principal, scope set, constraints, credential class, expiry, staleness — and nothing more. The resource owner decides what that permits. The service publishes the scope vocabulary as machine-readable metadata so consoles, SDKs and gateways agree on what exists, and stops there.

How it is realised on Google Cloud

The context is returned as a single structure the caller passes downstream without re-querying, carried as a signed header between the sidecar and the application. Product authorization is a separate concern in each service, reading that context. View 01 draws product authorization explicitly outside the boundary so the omission is a decision rather than an oversight.

Options weighed

• ChosenContext only; the resource owner decides: Keeps the hot path small and the blast radius of a product change local to that product.
• RejectedFull policy decision point: Every product's rules inside the most latency- and correctness-critical component on the platform, with every product change deploying to it.
• RejectedContext plus a coarse allow/deny on scope membership: Sounds harmless and is where a policy engine starts. The second request is always for a resource-level condition.
• Right elsewhereSeparate policy engine reading this context: Right for an organisation that wants centralised authorization. It should be a different service with a different SLO, consuming this one's output.

Consequences

What it buys

• The verification path stays a lookup, which is what keeps the p99 budget achievable.
• A product authorization bug cannot take down credential verification for the whole platform.
• The two failure modes stay distinguishable in logs and in incident response.

What it costs

• Every product implements its own authorization, with the inconsistency that implies.
• Customers see scopes from this service and permissions from products, and must reconcile the two mentally.
• There is no single place to answer 'what can this key actually do', which is a question customers reasonably ask.

Choose differently when

Fold authorization in when the platform is one product with one resource model. Split it out again the moment there are two, which will be before anyone expects.

Why it holds up over time

This boundary is what allows the platform's product surface to change completely over ten years without the credential service changing at all. It is also the boundary most often eroded by a well-meaning feature request, which is why it is drawn rather than described.

What shape is the scope vocabulary, who owns it, and may an entry ever be removed?

Context

Hierarchical scopes are expressive and become impossible to reason about at a glance, which matters because a developer picks scopes in a console in about eight seconds. Federated ownership scales with the organisation and reliably produces three spellings of 'read'. And removal is the quiet catastrophe: a scope withdrawn from the vocabulary breaks every customer integration holding a credential that carries it, with no warning and no migration path, at a time chosen by an internal team.

Decision

Scopes are flat `resource:action` strings, versioned, additive only, and centrally owned by one registry with one reviewer. Removals are rejected at the pipeline gate; a scope may be marked deprecated and stop being offered to new credentials, but it never stops resolving for credentials that already hold it. Each scope carries a grade — read, write or admin — which the staleness rule in ADR-07 consumes.

How it is realised on Google Cloud

The vocabulary is a declarative file, linted for synonyms and removals, built into an immutable signed artefact and rolled out in rings (ADR-20). A replay simulation runs the previous day's verifications against the candidate vocabulary and fails the build if any outcome changes. The registry is a service rather than a file at runtime, because the no-escalation check in ADR-14 needs the principal's own scopes at issuance time.

Options weighed

• ChosenFlat, additive, centrally owned: Legible in a console, safe for existing integrations, and coherent across product surfaces — at the cost of being a bottleneck.
• RejectedHierarchical scopes with wildcards: Expressive, and nobody can tell what payments:* will mean after the next product launch. A wildcard is a scope that grows without review.
• DeferredFederated per-service vocabularies: Scales with the organisation, and this is Core Architecture Question 7. The trigger to revisit is the rate of new-scope requests, not a complaint about review latency.
• RejectedFree-form scope strings: Maximum flexibility, zero ability to present a coherent consent screen or to grade a scope's consequence.

Consequences

What it buys

• A five-year-old customer integration keeps working, because nothing is ever removed.
• One vocabulary means a console, an SDK and a gateway describe the same thing the same way.
• Grading is available to the staleness rule without a separately maintained list.

What it costs

• The registry owner is on the critical path of every product team's roadmap, which is a real organisational cost and the reason ADR-13 has an open successor question.
• Additive-only means the vocabulary only grows; after ten years it carries scopes nobody remembers the purpose of.
• Flatness means some genuinely hierarchical resources are expressed clumsily.

Choose differently when

Federate when the number of product surfaces makes central review the dominant delay, and accept the drift — or invest in cross-vocabulary linting, which is the same central cost wearing different clothes.

Why it holds up over time

Additive-only is what makes the vocabulary safe to inherit. A future team can add, deprecate and grade without ever being able to break an existing customer, which is the property that lets the vocabulary outlive its authors.

What stops key creation from being a privilege-escalation path?

Context

If a principal with read access can create a credential carrying write scopes, then the read/write distinction is decorative: any compromised read-only session becomes a write-capable credential that outlives the session by a year. The check has to happen where the principal's own scopes are known, which is at issuance in the credential service — not in the console, which is one of several clients and can be bypassed by calling the management API directly.

Decision

Issuance refuses any scope the creating principal does not itself hold, evaluated in the issuance service against the principal's scopes as asserted by the platform IdP. Scope escalation on an existing credential requires step-up authentication and is recorded with its evidence in audit. The same rule applies to machine principals: a workload cannot mint a credential broader than the workload identity it runs as.

How it is realised on Google Cloud

The issuance service resolves the principal from the IdP assertion or the workload identity, queries the scope registry for the principal's held scopes, and computes the intersection. A request for anything outside it fails with an explicit error naming the missing scope, which is safe because the caller is authenticated. Step-up is required for escalation, constraint removal and bulk revocation, and the step-up evidence is written alongside the issuance record.

Options weighed

• ChosenRefuse at issuance against the principal's own scopes: Enforced at the one place that cannot be bypassed, and cheap because issuance is rare.
• RejectedCheck in the console only: Bypassed by calling the management API directly, which is documented and encouraged. A control in one client is not a control.
• DeferredAllow escalation with an approval workflow: Reasonable for an enterprise tenant with separation of duties, and it is a tenant-configurable extension rather than a change to the default.
• RejectedAllow any scope, detect escalation afterwards: Detection after issuance means the credential existed and may already be in a config file somewhere.

Consequences

What it buys

• A compromised low-privilege session cannot be laundered into a long-lived high-privilege credential.
• The rule is stated once, in one service, and applies to every client of the management API.
• Audit can answer 'who authorised this scope' years later, with the step-up evidence attached.

What it costs

• Issuance depends on the IdP and the scope registry being reachable, which is part of why the control plane's availability target is lower than verification's.
• A legitimate delegation — a CI system minting credentials for scopes it does not itself use — must be modelled as a principal holding those scopes, which is more setup than a wildcard would be.

Choose differently when

Relax only where every principal is already fully privileged, which usually means the tenant model is too coarse and that is the thing to fix instead.

Why it holds up over time

The rule is independent of how identity is represented; whatever replaces OIDC still yields a principal with a scope set. What ages is the enforcement point, and putting it in issuance rather than the console is what makes it survive the arrival of the fifth API client.

Should a credential that has not been used for ninety days expire automatically?

Context

Automatic retirement of idle credentials is genuinely valuable: it is the mechanism that keeps credential sprawl from becoming permanent. But making idleness an authorization input changes the nature of the hot path. Usage would have to be recorded durably, at 250,000 writes per second, before a verification could be trusted — and a lossy counter that can revoke something is a mechanism that will eventually revoke a credential that was in fact in use. The value of the feature is real; the cost lands on the one path that must stay a memory read.

Decision

last_used_at and call counts are observations: written asynchronously, aggregated in a write-optimised store, accurate to within 5 minutes, and explicitly lossy. Nothing reads them to make an authorization decision. Idle credentials are surfaced to their owners — in the console, in ownership review, and in rotation's report of a predecessor's last use — and a human decides. Automatic expiry on idleness is deferred to Phase 3 and, if built, must not read this counter.

How it is realised on Google Cloud

The sidecar emits a sampled usage event off the request path; aggregation writes to Bigtable on a rolling window. The console shows last-used timestamp and region against every credential, labelled as approximate. Rotation's report of the predecessor's last use carries the same label, and the automatic revocation at day 7 is driven by the clock, never by the counter.

Options weighed

• ChosenLossy observation, human decides: Keeps the hot path stateless and gives the owner what they actually need: evidence to retire a key with confidence.
• RejectedDurable usage write, automatic expiry on idleness: Makes verification a durable write at full request rate, and lets a dropped event revoke a credential that was in use.
• DeferredPeriodic sampling with a wide idleness threshold: A 180-day threshold makes the loss rate tolerable, which is the shape any future automatic retirement should take — with its own record.
• RejectedNo usage tracking at all: Cheapest, and it removes the evidence that lets anyone retire anything. Sprawl then becomes permanent.

Consequences

What it buys

• The verification path stays a read, with telemetry that may be dropped but may never delay an answer.
• Observability cost scales with traffic rather than with correctness requirements.
• The credential owner gets the one field that makes retirement a decision rather than a gamble.

What it costs

• Credential sprawl is bounded by quota and by human review, not by automatic retirement — slower and less complete.
• Approximate timestamps invite exactly the misuse this record forbids, so the labelling has to survive every future UI change.

Choose differently when

Durable usage tracking is right when verification volume is low enough that a write per request is affordable, or when regulation requires provable last-use rather than observed last-use.

Why it holds up over time

The separation between observation and input is the durable part. If automatic retirement is ever built, this record tells a future team exactly what it must not reuse, which is more useful than a prohibition with no reasoning attached.

How does a repository host tell the platform it has found a live credential, without the platform disclosing anything to an unauthenticated caller?

Context

Public-repository scanning only works if the credential is recognisable. That means publishing a stable, distinctive prefix — which is an intentional disclosure, and the right one: the alternative is a random string no scanner can find. The harder half is the confirmation step. A scanner that finds a candidate needs to know whether it is live, which means calling the platform. That caller cannot be authenticated in any meaningful per-customer way, and whatever it can ask, an attacker with a partial key can also ask.

Decision

Credential prefixes are stable, documented and distinctive per class and environment. The partner confirmation endpoint is unauthenticated by design, hard-quota'd per partner, and structurally incapable of returning anything but live-or-not: it never loads the tenant, the scopes or the owner into scope. Own-corpus scanning — internal repositories, logs, support attachments — is in the must-have tier alongside the partner integration.

How it is realised on Google Cloud

Cloud Armor in front of the endpoint with a per-partner quota of 20 requests per second, well below any useful search rate against 160 bits of entropy. The handler computes the digest, checks membership and returns a boolean; the code path has no access to the credential record's other columns. Sustained NOT_FOUND rates from a single source raise an alert regardless of quota compliance.

Options weighed

• ChosenUnauthenticated one-bit endpoint with a hard quota: Partners can act immediately, and the worst an abuser gains is confirmation of a key they already hold.
• RejectedAuthenticated partner integration: Cleaner on paper. Partners scan for hundreds of vendors and will not hold per-vendor credentials, so the integration does not happen.
• DeferredPlatform pulls from partner feeds instead: Removes the endpoint entirely and adds latency plus a dependency on each partner's feed format. Worth it only if the endpoint proves abusable.
• RejectedNo prefix — random credential strings: Discloses nothing and makes the platform undetectable to every scanner, which trades a theoretical disclosure for a real detection capability.

Consequences

What it buys

• Leaked credentials are found by parties the platform does not have to build or operate.
• The prefix makes a credential legible in a log, an error and a support ticket without exposing the secret.
• The endpoint's blast radius is one bit, enforced by what the code path can reach rather than by a filter on the response.

What it costs

• The prefix tells an attacker what they have found, which marginally increases the value of a leak they already hold.
• An unauthenticated endpoint on the public internet is permanent attack surface that must be watched.
• Partner coverage is uneven: keys leaked to private repositories, logs and tickets are invisible to it, which is why own-corpus scanning is not optional.

Choose differently when

Pull from partner feeds instead when the endpoint shows abuse that quotas cannot contain, or when a regulator objects to an unauthenticated interface regardless of what it returns.

Why it holds up over time

The prefix is the most permanent artefact this platform produces: it appears in customer configuration everywhere and can never be changed without breaking scanners. The one-bit constraint is what keeps that permanence safe.

What is the platform allowed to do on its own when it believes a credential is compromised?

Context

A confirmed partner match is near-certain evidence: the exact secret was found in a public commit. Acting automatically there is straightforwardly right — waiting for a human costs minutes during which the credential is public. An anomaly signal is different in kind. First use from a new country is a deployment to a new region. A tenfold rate change is a successful product launch. An unexercised scope is a new feature. Revoking on those takes down a live customer integration for a reason the customer will find absurd, and the trust cost of one such event exceeds the leak it prevented.

Decision

On a confirmed leak, the configured response executes automatically within the leak-response SLO: notify, and either revoke, downgrade to read-only scopes, or quarantine pending owner confirmation, per tenant policy and credential class. Automated action on anomaly signals alone is opt-in per tenant. Every automated action is reversible and carries its full decision trail, including what triggered it.

How it is realised on Google Cloud

The leak-response engine reads a per-tenant policy keyed by credential class. Downgrade is implemented as a scope-set change on a new credential version rather than a revocation, so it is instantly reversible. The trigger trail — source, evidence, policy version, timestamp — is written to the incident record, and the lifecycle webhook carries the same identifiers so the customer's own systems can act.

Options weighed

• ChosenAuto on confirmed, opt-in on anomaly: Matches the action's aggressiveness to the evidence's certainty, and puts the false-positive decision with the party that bears it.
• RejectedAuto-revoke on any signal: Fastest containment and the fastest route to a customer outage caused by a heuristic. One such event costs more trust than several slow containments.
• RejectedHuman confirmation for everything: Safe against false positives and adds minutes during which a public credential is live. The 60-second SLO is not achievable with a human in the loop.
• DeferredDowngrade to read-only as the universal default: An attractive middle for platforms where read access is not itself sensitive. Not here, where a read scope can reach customer financial data.

Consequences

What it buys

• Confirmed leaks are contained in under a minute without waking anyone.
• The tenant owns the risk appetite for uncertain signals, which is where the consequence lands.
• A disputed automated action has an answer, because the trigger trail is retained.

What it costs

• Anomaly detection is off for most tenants, so the signals it produces are advisory and slower to act on.
• Per-tenant policy is configuration that must be correct, discoverable and defaulted sensibly, or it becomes a support burden.
• Downgrade-as-a-version-change is a second containment mechanism with its own semantics to test.

Choose differently when

Auto-act on anomalies when the platform's credentials authorise only low-consequence actions, or when the customer base is internal and an incorrect revocation is a five-minute inconvenience rather than an outage.

Why it holds up over time

Detection techniques will improve and the confidence threshold will move. What should not move is the principle that the aggressiveness of an automated action is tied to the certainty of its evidence, and that the party bearing the false-positive cost chooses the setting.

What is the topology, and does every component need the same recovery objective?

Context

Treating the whole service as one thing produces either an over-engineered control plane or an under-engineered data plane. The two halves have genuinely different requirements: verification failing for one second is visible to every customer of the platform, while issuance failing for fifteen minutes is an inconvenience a developer works around. Building both to the same standard means paying for the harder one twice, or — more commonly — settling for the easier one everywhere.

Decision

Verification is active-active in all three regions with RTO 0: losing a region is a capacity event, not an availability event. The control plane runs a leader region with warm standbys at RTO ≤ 15 minutes. Availability targets differ accordingly: ≥ 99.99% for verification, ≥ 99.9% for the management API. The two are deployed, released and alerted on separately.

How it is realised on Google Cloud

GKE in europe-west1, us-east4 and asia-southeast1, each running the product fleet with its sidecars and a regional Memorystore and snapshot bucket. Control-plane workloads run in all three but accept writes only in the leader. Spanner is multi-region rather than per-region because the revocation log's RPO 0 requires a quorum that spans regions; a single-region configuration would make the most important guarantee in the architecture regional.

Options weighed

• ChosenActive-active data plane, single-leader control plane: Pays for the hard guarantee where it is needed and takes the cheaper one where it is survivable.
• RejectedActive-active everywhere: Multi-master issuance means conflict resolution on credential creation and quota, for a write path measured in hundreds per second. Cost with no benefit.
• RejectedSingle region for everything: Makes regional loss a total outage of verification, which is the one thing that must never stop.
• RejectedRegional Spanner with async cross-region replication: Materially cheaper, and revocation durability becomes regional. That is the guarantee the multi-region cost exists to buy.

Consequences

What it buys

• Regional loss costs capacity, not availability, for the path every request depends on.
• The control plane can be simpler, and its failover exercised without touching verification.
• Two clearly different SLOs make it obvious which alerts wake someone at 3 a.m.

What it costs

• Multi-region Spanner is the single largest line in the architecture, justified by exactly one requirement.
• Two deployment topologies and two failover procedures to maintain and rehearse.
• Control-plane failover, being rare, is the procedure most likely to have rotted when it is needed.

Choose differently when

Single region is right for a platform serving one geography with a regional availability target. Active-active issuance becomes right if credential creation ever becomes a high-rate machine operation rather than a human one.

Why it holds up over time

The split between a plane that must never stop and a plane that may pause is the durable structure. Regions, services and even clouds can be swapped underneath it; what would break it is letting an issuance-path dependency creep into verification, which view 20 draws as a refused crossing for that reason.

How does anyone know that revocation still works today?

Context

Every component in the propagation chain can fail silently. A subscriber that stops applying messages reports healthy. A snapshot builder that writes a stale object reports success. A staleness guard with an inverted comparison passes its unit tests and fails in production in exactly one direction. None of these produce errors, and all of them mean a revoked credential keeps working. Instrumenting the feed does not help: the feed will report that it published, which is true and irrelevant.

Decision

A canary continuously issues a synthetic credential, verifies it, revokes it and re-verifies it, end to end, through the same paths a real credential takes. Propagation p99 is the measurement from that loop. The canary runs in a separate Google Cloud project, in a region chosen independently of the leader, and the platform's own operators cannot deploy to it. Its silence is alerted on by a dead-man check outside both.

How it is realised on Google Cloud

A Cloud Run job in the assurance project runs the cycle every 60 seconds against each region's verification points, recording the interval between revocation acknowledgement and first denial at each point. A dead-man timer external to both projects fires if the canary itself stops reporting, because a stopped canary and a healthy platform produce the same absence of alerts.

Options weighed

• ChosenEnd-to-end canary, externally hosted, dead-man alerted: The only signal that catches a silent propagation failure, and the only one whose own failure is also detected.
• RejectedFeed instrumentation and subscriber lag metrics: Necessary and insufficient. A subscriber that receives and fails to apply reports zero lag while honouring revoked credentials.
• RejectedPeriodic manual drill: Proves it worked on the day of the drill. The failures being guarded against appear between drills and persist.
• RejectedCanary inside the platform's own project: Cheaper, and it shares a failure domain with the thing it is judging — including the deploy that broke it.

Consequences

What it buys

• The propagation SLO is a measurement rather than a claim, and a regression is visible within a minute.
• A change that silently breaks the staleness guard or the overlay is caught by the canary rather than by a customer.
• Independence means an outage of the platform does not take the witness with it.

What it costs

• A second project, with its own access model, budget and on-call ownership.
• The canary writes real credentials continuously, which is a small permanent load and a set of synthetic records to exclude from every report.
• Its own reliability now matters, which is what the dead-man check is for and one more thing to maintain.

Choose differently when

An in-platform canary is acceptable when there is no separate project available and the alternative is no canary. Never accept feed metrics alone as the assurance for a revocation guarantee.

Why it holds up over time

The canary tests the contract rather than the implementation, so it survives replacing the bus, the store, the sidecar and the cloud. It is the single artefact most likely to still be valuable in ten years, and the one that keeps the architecture's central claim honest.

How does a scope vocabulary or policy change reach every verification point without being able to break them all at once?

Context

Configuration is the fastest path to a total outage in this architecture, because a verification point applies it immediately and there are thousands of them. A bad vocabulary or a malformed policy reaches all of them within seconds of publication, and the natural failure behaviour of a config parser — fall back to a default — is catastrophic here, because the permissive default is 'allow'. The same change that would be a slow, reviewable code deploy is, as configuration, an instant fleet-wide event.

Decision

Configuration is built into immutable, signed artefacts with provenance, not pushed live from a database. Rollout is staged: a canary ring of three services for thirty minutes with an error-rate watch and automatic rollback, then an internal ring, then the remainder. A verification point that cannot parse an artefact keeps the last good one and reports its version; it never falls back to a permissive default. Config version spread across the fleet is a monitored quantity.

How it is realised on Google Cloud

The artefact is built in the same pipeline as the vocabulary lint and the replay simulation (ADR-13), signed, and published to Cloud Storage with its provenance. Verification points pin the last good version locally, so a restart during a bad rollout comes back on the last known-good rather than on the broken artefact. The rollback is a single command that re-pins a version, not a rebuild.

Options weighed

• ChosenImmutable artefacts, ring rollout, refuse-and-pin: Makes the blast radius of a bad config a ring rather than the fleet, and makes the failure mode 'stale' rather than 'permissive'.
• RejectedLive config from a database: Simplest and fastest, with no staging, no rollback and no provenance. The fastest available way to take the platform down.
• RejectedConfig shipped in the application binary: Safest rollout there is, and a scope addition now waits for a full deployment cycle across 180 services.
• RejectedParse-fail to a permissive default: The conventional behaviour, and here it converts a syntax error into an authorization bypass.

Consequences

What it buys

• A poison config is contained to a ring and rolled back automatically.
• Restart during a bad rollout is safe, because the last good version is pinned locally.
• Version spread across the fleet makes an incomplete rollout visible instead of assumed.

What it costs

• A scope addition takes hours to reach the whole fleet rather than seconds, which is a real friction for product teams.
• Artefact building, signing and provenance is a pipeline to own.
• Pinned last-good versions mean a point that has been isolated for a long time is running old policy, which is a second staleness to reason about.

Choose differently when

Live config is acceptable where a bad value degrades one feature rather than the authorization decision. The distinction is whether the config can make the system permissive, not how quickly it needs to change.

Why it holds up over time

Immutability and staged rollout are stable practices, and the refuse-rather-than-default rule is the part most likely to be quietly reversed by a future engineer fixing a parsing bug. It is recorded here with its reasoning so that reversal is a decision rather than a patch.