Internal Developer Platform

Internal Developer Platform · Solution Architecture v1.0 · Microsoft Azure · Platform Architecture · 2026-09 · 21 views · 16 architecture decision records

The paved road is a product teams may decline; the guardrails are controls they cannot. Generation and enforcement are separate planes, and they meet only at the runtime.

Creating a new service at this organisation takes nineteen working days and touches seven teams: a repository request, a pipeline request, a namespace request, a database request, a DNS and certificate request, a secrets onboarding request, and a security review that arrives after the architecture is already built. Every step is a form filled in by an engineer who does not want to be an expert in the thing the form is about, and read by a specialist who is the bottleneck for four hundred other engineers. The cost is not only the nineteen days. It is the services that never get split because splitting is expensive, the security review that lands too late to change anything, and the gradual accumulation of workloads nobody can attribute to a team. The platform's job is to make the distance between an engineer's intent and a production-grade running system as short as it can honestly be — without making the result less safe, and without becoming a ticket queue with a nicer interface.

An engineer picks a golden path in the portal or the CLI. The scaffolder creates the repository with its catalog descriptor, deployment intent, infrastructure intent, a reference to the shared pipeline, and the observability and on-call wiring already in place. Everything the engineer declares lives in their own repository; the platform's surfaces write intent and then stop. Two reconcilers converge the world on that declaration — one for workloads into a shared AKS fleet, one for infrastructure through Terraform against Azure Resource Manager — and both are idempotent, resumable and drift-detecting. Alongside them, and sharing no component with them, runs the enforcement plane: policy authored as code, signed into a bundle, cached locally at every evaluation point, advisory in the pull request and binding at admission into the cluster. A component that never touched a golden path meets exactly the same binding checks. Everything that results — what is running, who owns it, what it cost, what it is failing — is projected into a catalog that is rebuildable rather than authoritative, and into the adoption and DORA metrics by which the platform judges itself.

What it is, and what it is not

• A product engineers choose because it is faster than the alternative — not a mandated standard whose adoption figure is 100% by definition and therefore says nothing.
• A set of controls enforced where workloads run — not a template that happens to contain the right settings until someone forks it.
• A mechanism that reconciles towards intent the team owns — not a database of other people's configuration with a form in front of it.
• A system that may fail without production noticing — not a portal in the path of anyone's deploy, rollback or running traffic.
• An answer that arrives in minutes and explains its own refusals — not a ticket queue with better typography.

The decisions that are the architecture

1. Guardrails live in the runtime, not the template (ADR-05) — Enforcement is a property of admission into the cluster, authored and signed separately from any golden path. A fork removes the template and never the control.
2. One policy source, two evaluation points (ADR-06) — Advisory in the pull request to teach, binding at admission to hold. A CI-pass-then-admission-fail is defined as a platform defect and measured as one.
3. Intent lives in the team's repository (ADR-01) — The portal writes intent and owns nothing. The platform's catalog is a projection, which is what lets it be rebuilt rather than protected at RPO 0.
4. Provision by reconciliation (ADR-02) — Declared state converged on continuously, so a partial failure resumes and a console edit becomes a detected event rather than invisible drift.
5. Track the template by default; forking is recorded (ADR-03) — The fork ratio is the number that predicts migration cost, and the pipeline is consumed by reference so the most change-prone artefact is not part of the fork surface.
6. Attribution at creation (ADR-11) — Owner, cost centre, tier and data classification are enforced properties applied when a thing is made, because nothing acquires an owner afterwards.
7. Entitlement derives from ownership (ADR-10) — Catalog ownership resolves to a live identity group, so a reorganisation moves access with the components and there is no second list to decay.
8. The platform is never in the critical path (ADR-12) — A portal outage costs the ability to change, never the ability to serve. Running workloads are never evicted because the platform cannot evaluate them.
9. Exceptions are named, approved by another, expiring and published (ADR-08) — Relief always exists; making it visible is what stops it becoming a quiet policy edit for the loudest team.
10. A thin abstraction with a designed escape hatch (ADR-07) — Every concept the golden path hides is met for the first time at 03:00, so the substrate stays addressable and access to it is an entitlement rather than an incident favour.

Why this should still be right in ten years

An internal platform outlives the tools it was built from — the portal, the reconcilers, probably the cloud. These are the properties that should survive all of that, and the ones a future team would most easily discard without noticing what they were for.

• The plane separation names no technology. ADR-05 says that evidence of a control must come from where the workload runs, not from how it was created. That is true of Kubernetes admission, of whatever replaces it, and of a substrate that has no admission point at all — in which case the architecture must say honestly that its coverage claim has weakened, rather than quietly keep making it.
• Ownership of truth decides ownership of the system. Keeping intent in the team's repository is not a Git preference. It is the reason the platform can be rebuilt without asking 140 teams what they had, and the reason the platform team is never asked to act as the owner of somebody else's configuration on the worst day.
• Optional adoption keeps the feedback signal alive. A mandate would raise coverage and destroy the only honest measurement of whether the paved road is good. The shadow estate is uncomfortable to report and is the most valuable number the platform produces.
• Cognitive load is a budget that will be spent if it is not defended. Every golden path will want one more concept, and each addition is individually reasonable. Stating the budget as a design constraint is what makes the tenth addition arguable rather than inevitable.
• The control plane stays out of the data path. The rule that a platform outage may cost the ability to change but never the ability to serve is the one that decides whether the platform survives its first bad day. It is also the first rule broken under deadline, which is why the SLOs are published to tenants.
• What is rebuildable and what is not is a permanent question. The storage classification in ADR-16 is a method rather than a configuration: ask of every store what it could be reconstructed from, and protect only the ones with no answer. That survives any change of database, cloud or retention regime.

Non-functional targets

Every number here is a stated assumption from the requirement, chosen to be argued with rather than believed. The right-hand column names the view where the mechanism that meets it is drawn.

Scope

In scope

• A software catalog with enforced ownership, lifecycle state and tier, reconciled against the runtime in both directions.
• Golden paths that scaffold repository, pipeline reference, manifests, observability wiring and catalog descriptor in one action, with drift measurement and automated update pull requests.
• Self-service provisioning by declared intent for a bounded catalogue of resource kinds, admitted against quota, budget and data classification.
• A shared build and deploy path producing signed, attested, content-addressed artefacts promoted unchanged through environments, with progressive delivery and pipeline-independent rollback.
• Policy-as-code guardrails evaluated advisorily in the pull request and bindingly at admission, with a named, expiring exception register.
• Portal, CLI and API over one behaviour surface, with a single component page carrying health, deploys, dependencies, on-call, cost and open findings.
• Workload identity everywhere, brokered secrets, and an append-only audit of every privileged action.
• Adoption, DORA and control-coverage reporting, including the shadow estate.

Explicitly out of scope

• Product code, product architecture and product on-call. The platform paves the road; the teams drive on it.
• The landing zone: subscription vending, network topology and the organisation's Azure Policy floor. The platform is its tenant.
• Being an approval workflow. A request the platform accepts and then queues for a human has already failed at its only job.
• Owning tenant multi-region strategy — the platform's paired region carries its own recovery, not an active copy of every tenant.
• Per-pull-request ephemeral environments and self-service golden-path authoring, both deferred to Phase 3.

What a six-week prototype should prove

Four of this architecture's claims are cheap to test and expensive to be wrong about. A prototype that proves these can be built on; one that only demonstrates scaffolding a repository has proved the easy half.

1. Plane separation end to end: scaffold a component, fork its template so every generated control is deleted, deploy it, and confirm the binding admission checks still refuse it for exactly the same reasons. Then deploy a vendor Helm chart that was never scaffolded and confirm it meets the same checks.
2. One policy source, two points: take a real control, evaluate it in the pull request and at admission from the same signed bundle, then deliberately desynchronise the bundle and confirm the divergence is detected and reported rather than silently tolerated.
3. Reconciliation under failure: kill a provisioning run halfway through a multi-resource set, confirm it resumes from the declaration without a human reconstructing state, then change a resource in the Azure console and confirm the drift is detected, attributed and reported within the lag budget.
4. The migration lever: scaffold twenty components, publish a template change, and measure what fraction land automatically, how long the tail takes, and what it costs a platform engineer per forked component. That ratio, more than any latency number, decides whether twelve people can serve nine hundred.

• Fork a scaffolded component's template so every generated control is deleted, deploy it, and confirm admission refuses it for exactly the same reasons — then deploy a vendor Helm chart that was never scaffolded and confirm it meets the same checks.
• Desynchronise the policy bundle between the pipeline evaluator and the admission webhook, and confirm the divergence is detected and reported rather than experienced by an engineer on a Friday.
• Kill a provisioning run halfway through a multi-resource set and confirm it resumes from the declaration, with no human reconstructing which half exists.
• Change a provisioned resource in the Azure console and confirm the drift is detected, attributed to the owning team and reported inside the lag budget.
• Take the portal down entirely and confirm a deploy, a rollback and a reconciliation all still complete.
• Delete the identity group that owns a running component and confirm it is quarantined and reported rather than continuing to run unattributed.
• Publish a template change across twenty scaffolded components and measure what lands automatically, how long the tail takes, and the platform-engineer cost per forked component.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 16 records across 6 areas, each with the alternatives that lost and what the choice costs.

Internal Developer Platform · Solution Architecture v1.0 · Microsoft Azure · Platform Architecture · 2026-09 · 21 views · 16 architecture decision records

The argument these decisions serve is summarised in the Architecture One-Pager.

Sixteen decisions make up this architecture. Everything else across the twenty-one views is convention, sizing or consequence. Each record states the forcing question, the context that makes it hard, what was decided, how it is realised on Azure, the options weighed, what the choice buys and costs, the conditions that would flip it, why it should still be right in ten years, and the lesson that transfers beyond this platform.

Status of this document. This is a design, not a report on a running system. The rates, latencies, volumes, retentions and targets are the requirement's stated assumptions for a mid-size engineering organisation — 900 engineers, 140 teams, 1,400 components, 5,200 deploys a week, a platform team of twelve — invented to be defensible and arguable rather than absent. They are to be replaced by measured telemetry before build, and four in particular are the first numbers real data should overturn: the 19-working-day baseline the whole business case rests on, the 16% fork ratio that predicts migration cost, the ≥ 70% adoption target at month 12, and the assumption that twelve platform engineers can serve nine hundred.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it is realised on Google Cloud: The concrete mechanism: which service or package, configured how, in which project.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

The two planes: Why generation and enforcement are separate machinery, and where each one binds.

• ADR-05 · Guardrails are enforced at the runtime, never inside the templates that generate components
• ADR-06 · One policy source, two evaluation points; fail closed on new admissions and open on running workloads

Intent and reconciliation: Where truth about what should exist lives, and how the world is moved towards it.

• ADR-01 · Intent lives in the team's repository; the portal writes it and owns nothing
• ADR-02 · Provision by reconciling declared intent, not by calling the cloud from the portal
• ADR-04 · Quota, budget and data classification are checked at admission, and a rejection is written for the requester

Knowledge and attribution: What the platform knows about the estate, and how every running thing keeps an owner.

• ADR-09 · The catalog is a projection, reconciled against the runtime in both directions
• ADR-10 · Ownership resolves to a live identity group, and entitlement derives from it
• ADR-11 · Owner, cost centre, tier and classification are applied at creation as enforced properties
• ADR-16 · Storage is classified by what a loss would cost, and only one band carries RPO 0

Identity, secrets and the control plane: Who may do what, and the fact that the platform is the most privileged thing in the estate.

• ADR-08 · Exceptions are named, approved by someone else, expiring and published
• ADR-15 · Workload identity everywhere, and the control plane is held to the controls it enforces

The runtime and the fleet: The tenancy boundary, and recovery that does not depend on the machinery that broke.

• ADR-13 · Rollback does not depend on the pipeline that produced the deploy
• ADR-14 · Namespace per component in shared clusters, with staged fleet upgrades and a canary component per path

Platform as a product: The decisions that only make sense if teams are allowed to say no.

• ADR-03 · Components track their golden-path template by default, and forking is a recorded, supported choice
• ADR-07 · A thin abstraction with strong defaults, and an escape hatch that is a designed path
• ADR-12 · The API is primary, and no platform component sits in the critical path of running production

Technology by capability

The platform is built on Microsoft Azure with open-source platform components running on it. Azure was chosen for rotation — the four use cases before this one ran on Google Cloud and AWS — and because an internal developer platform's two hardest questions, who is entitled to provision what and where the isolation boundary sits, map onto Entra entitlements and the management-group hierarchy rather than having to be invented on top of a flat account model. Everything in the requirement document outside its first section is vendor-neutral, and the table below is where the cloud is actually chosen.

The decisions, and the alternatives that lost

The two planes

Why generation and enforcement are separate machinery, and where each one binds.

ADR-05 · Guardrails are enforced at the runtime, never inside the templates that generate components

Status: Accepted · Shown on views: 07, 08, 15, 18

Is a control something a golden path produces, or something the runtime insists on?

Context. The natural build puts the controls in the template: the pipeline that scans the image, the manifest with the right security context, the module with the right network rules. One artefact, one review, one place to change things — and on day one every component is generated from it and every component is compliant. The failure arrives about nine months later. A team forks the template to do something reasonable it does not support. Another deploys a vendor Helm chart that was never scaffolded. A third pins an old version because the new one broke their build. None of these is misbehaviour; each is the normal cost of a platform real teams use. But each silently removes the controls along with the template, and the platform cannot know, because the only evidence a control was ever in force was that a template produced it. The estate's compliance figure is now a statement about how components were created, not how they are running.

Decision. Enforcement is a property of the runtime. Guardrails are authored as policy-as-code, versioned and signed independently of templates, and evaluated at admission into the cluster. They apply identically to a component the platform scaffolded and to one that never touched a golden path. No template, fork, portal action or pipeline change can remove a binding control.

How it is realised on Google Cloud. Policy in its own reviewed repository, built into a signed OCI bundle in ACR, pulled and cached by admission webhooks running in every cluster. The minimum binding set: workload identity present and no static cloud credentials; image from a trusted registry with a valid attestation and no unmitigated critical vulnerability; resource requests and limits declared; owner and cost tags present; no unauthenticated ingress for an internal component; storage region appropriate to the declared data class.

What it buys

• Coverage is measurable against the running estate: ≥ 99% of production workloads, off-road ones included.
• The golden path is freed to be judged purely on whether it is good, which is what makes the adoption number meaningful (ADR-12).
• A control change ships without touching 1,400 repositories.

What it costs

• Policy and templates must be kept coherent by discipline rather than by being the same file.
• The runtime becomes a harder dependency: a broken policy bundle can stop every deploy in the fleet (ADR-06).
• Standing pressure to soften a policy for the loudest team, answered by the exception register rather than by a quiet edit (ADR-08).

Choose differently when. In an organisation where every deploy provably passes one pipeline and no other path exists, pipeline enforcement is equivalent and cheaper. If a future runtime offered no admission interception point at all, enforcement would have to move to the provisioning boundary — and the platform would have to say honestly that it no longer covers what it used to.

Why it holds up over time. This is a statement about where evidence of a control comes from, not about Kubernetes. Any substrate with an admission decision can carry it; any substrate without one forces the question to be answered again, honestly.

Lesson. A control you can only demonstrate by pointing at how something was created is not a control. Enforce where the thing runs, because that is the only place that is still true a year later.

ADR-06 · One policy source, two evaluation points; fail closed on new admissions and open on running workloads

Status: Accepted · Shown on views: 02, 13, 15, 18

If the unbypassable check happens at admission, what is the point of checking in the pull request — and what happens when the evaluator is unavailable?

Context. Admission is the right place to be binding and the worst place to be informative: the verdict arrives on a Friday afternoon, after the build, in a message written for a platform engineer. Checking in the pull request is fast, teachable and trivially bypassed. Doing both is obviously right and introduces the failure that destroys trust faster than having no pull-request check at all — a control that passes CI and fails at admission. The availability question is the same decision viewed from the other side: a policy evaluator that fails closed on everything can evict a healthy production estate, and one that fails open admits unevaluated change.

Decision. Both evaluation points are driven from one signed policy bundle. Advisory evaluation in the pull request and pipeline exists to explain and to be useful; binding evaluation at admission exists to be unbypassable. A divergence between the two is defined as a platform defect and instrumented as such. When the evaluator cannot reach a current bundle it uses the last known good one; when it cannot evaluate at all it fails closed for new and changed admissions and open for already-running workloads.

How it is realised on Google Cloud. Signed bundle in ACR, pulled and cached locally by three webhook replicas per cluster and by the pipeline evaluator, so evaluation never requires a call to a central service on the deploy path. Admission budget ≤ 250 ms p99. The CI-pass-then-admission-fail rate is a first-class metric in view 18.

What it buys

• Engineers meet a control in the pull request, where it can teach, and the floor still holds at admission.
• A cached local bundle means a control-plane outage degrades authoring, not enforcement.
• The asymmetric failure mode is explicit, so nobody has to guess what happens during an incident.

What it costs

• The admission webhook is now in the path of every workload admission across the fleet, with the availability obligation that implies (≥ 99.95%).
• A bad policy bundle is a fleet-wide incident; publishing is staged and reversible for that reason.
• Keeping advisory and binding verdicts identical is continuous work, not a property that holds by itself.

Choose differently when. If divergence between the two points proved impossible to eliminate — different inputs available at different stages, say — the honest response is to narrow the advisory set to the checks that can be evaluated identically, rather than keep a check that lies. If admission latency could not be held under budget, evaluation would have to move to a pre-admission controller with a reconciliation sweep, accepting a window of unevaluated state.

Why it holds up over time. Local evaluation from a signed, distributed artefact is how policy scales anywhere, because it removes a network call from the critical path and removes a central service from the blast radius. That outlives any particular engine.

Lesson. Two checks are only better than one if they can never disagree. The moment they can, the friendly one stops being believed and the strict one becomes the only one anybody reads.

Intent and reconciliation

Where truth about what should exist lives, and how the world is moved towards it.

ADR-01 · Intent lives in the team's repository; the portal writes it and owns nothing

Status: Accepted · Shown on views: 02, 07, 10, 11

When an engineer asks the platform for something, where does the record of what they asked for live?

Context. The obvious design stores the request in the platform's own database: the portal is a form, the database is the system of record, and the platform reconciles the world towards its own rows. It is simple, it gives synchronous feedback, and it quietly makes the platform the owner of every team's configuration. The consequences arrive later. A team cannot review a change to their own infrastructure in the same place they review their code. A platform outage becomes an outage of the ability to describe anything. Restoring the platform's database restores somebody else's intent from a backup they never saw. And the one artefact that should be readable, diffable and blameable by the team who owns it is inside a system they cannot query.

Decision. The catalog descriptor, the deployment intent and the infrastructure intent are files in the component's own repository. The platform's surfaces — portal, CLI, API — write those files, through a commit or a pull request, and then stop. The platform's own stores hold projections, provisioning state and evidence, never the declaration of what should exist.

How it is realised on Google Cloud. A scaffolded repository contains catalog-info.yaml, a deployment manifest directory reconciled by Argo CD, and a Terraform intent file consumed by the infrastructure reconciler. The platform API commits to those paths using a GitHub App installation scoped to the component's repository. Portal actions that change infrastructure produce a pull request, not a write.

What it buys

• Review, history, blame and revert for infrastructure use the tooling every engineer already knows.
• The platform's catalog becomes a rebuildable projection (ADR-09), which is what lets it carry a four-hour RTO instead of a zero-loss obligation.
• An audit answer — what was this component allowed to be, on that date — is a Git history question rather than a platform database restore.

What it costs

• Every platform action is asynchronous from the engineer's point of view: submitted, converging, converged.
• The platform depends on the availability of the source control system for authoring, and must degrade clearly when it is unavailable.
• Bulk changes across 1,400 repositories become 1,400 pull requests, which is the migration cost ADR-03 has to carry.

Choose differently when. If the organisation moved to a monorepo with a single reviewed configuration tree, the repository boundary would stop being the team boundary and the decision would need restating — the principle survives, the file layout does not. If source control were unavailable often enough to block authoring, a durable write-ahead queue in front of the commit would be the first concession, not a move of truth.

Why it holds up over time. Nothing in this decision names Git, GitHub or Terraform. It says that the description of a system belongs to the team that owns the system, and that the platform is a mechanism rather than an owner. That survives a change of source control, of cloud, and of the platform team.

Lesson. Whoever holds the source of truth owns the thing. If a platform holds the description of every team's system, it has quietly taken ownership of all of them, and it will be asked to act like an owner on the worst day.

ADR-02 · Provision by reconciling declared intent, not by calling the cloud from the portal

Status: Accepted · Shown on views: 10, 14, 16

Does a provisioning request execute an imperative sequence of cloud API calls, or declare a desired state that a controller converges on?

Context. An imperative portal is faster to build and immediately satisfying: press the button, watch the resources appear, get an error you can read. It fails in three specific ways at scale. A run that dies halfway leaves a half-built resource set and no record of which half, so somebody reconstructs state by hand. A change made in the cloud console is invisible, because the portal has no opinion about what should exist — only about what it did. And a retry is dangerous rather than boring, because the operation was never idempotent.

Decision. Provisioning requests are written as declarative intent (ADR-01) and reconciled continuously by controllers that hold state and locks. Every apply is idempotent and resumable; drift between declared and actual is detected, reported to the owning team, and either reverted or recorded as an exception.

How it is realised on Google Cloud. Terraform modules from a versioned internal registry, executed by a runner pool on AKS with workload identity, against state in an Azure Storage account with locking. Drift detection re-plans on a schedule and on resource-graph change events, with a target reconciliation lag of ≤ 5 minutes p95.

What it buys

• A partially failed provision is resumed from the same declaration rather than reconstructed.
• Out-of-band changes become detectable events instead of permanent invisible divergence.
• The same mechanism serves creation, change and de-provisioning, so there is one path to operate rather than three.

What it costs

• Latency: an engineer sees 'converging' rather than 'created', and the 12-minute p95 target has to be defended.
• A stuck reconciliation is harder for a non-specialist to debug than a failed API call, so the platform must explain convergence state in the engineer's terms.
• State becomes the most dangerous object in the estate, which is why it carries RPO 0 and an exercised restore (ADR-16).

Choose differently when. For a genuinely trivial resource kind with no dependents — a DNS record, say — an imperative path is defensible and the indirection buys nothing. If the estate were small enough that drift never happened and a human could reconstruct any failure in ten minutes, this machinery would be overhead.

Why it holds up over time. Reconciliation towards a declaration is the same idea whether the controller is Terraform, Crossplane, an operator or something not yet written, and whether the target is Azure or not. The decision constrains the shape of the system, not its tools.

Lesson. The difference between a portal and a platform is what happens on the failure. A portal tells you the call failed; a platform already knows what should exist and keeps trying.

ADR-04 · Quota, budget and data classification are checked at admission, and a rejection is written for the requester

Status: Accepted · Shown on views: 04, 14

When a request cannot be satisfied, when does the requester find out, and in whose language?

Context. Cloud limits are real and arrive as provider errors halfway through an apply: a subscription quota, a regional capacity limit, a policy assignment the landing zone set. An engineer who meets one of these mid-provision has a half-built resource set and an error message written for a cloud operator. That moment is the trough of the journey in view 04, and it is where a self-service platform loses people back to tickets — not because the answer was no, but because the answer was unintelligible.

Decision. Quota, budget and data-classification checks run at admission, before any resource is created. A rejection names the constraint in the team's terms — which quota, how much is in use, who can raise it — and the request is never partially executed. Classification determines permitted regions and storage, and is checked against the declared data class rather than inferred.

How it is realised on Google Cloud. Admission checks run in the pull-request evaluation and again in the reconciler before plan. Quota state is tracked per team per environment; subscription headroom is monitored continuously so that a provider limit is anticipated rather than discovered.

What it buys

• No partially created resource sets from a foreseeable limit.
• Budget enforcement happens before spend rather than in a monthly report.
• An engineer can resolve their own rejection, which is what the ≥ 60% self-resolution target in view 18 measures.

What it costs

• The platform must model quota accurately enough to be trusted, and a wrong rejection is worse than none.
• Tracking headroom across subscriptions is ongoing work that grows with the estate.

Choose differently when. If provider quota errors became rare and cheap to recover from, attempt-and-report would be defensible. If the estate were small enough that one subscription served everything, headroom tracking would be a dashboard rather than a control.

Why it holds up over time. Every platform that hands out constrained resources eventually learns that the refusal is part of the product. That is independent of cloud, quota model and era.

Lesson. Design the rejection as carefully as the success path. Users forgive a no; they do not forgive a no they cannot act on.

Knowledge and attribution

What the platform knows about the estate, and how every running thing keeps an owner.

ADR-09 · The catalog is a projection, reconciled against the runtime in both directions

Status: Accepted · Shown on views: 02, 10, 11, 12

Is the platform's picture of the estate built from what teams declared, or from what is actually running?

Context. A catalog built from declarations is intentional, reviewable and always somewhat wrong, because reality drifts. One discovered from the runtime is always accurate about the present and cannot express ownership, intent, or anything not yet deployed. Holding both forces a rule about who wins when they disagree, and that rule is an architectural decision rather than a data-quality preference: it decides whether an undeclared running workload is an error or simply a fact.

Decision. The catalog is a projection over declared intent and observed runtime state, rebuildable from both. Divergence is reported in both directions — a declared component with nothing running, and a running workload with no catalog entry — and neither is silently resolved. Declaration wins for ownership and intent; observation wins for what exists now.

How it is realised on Google Cloud. Descriptor ingestion is webhook-driven with a periodic full reconcile as the backstop. A runtime reconciler reads cluster state and the Azure resource graph. Catalog freshness target ≤ 10 minutes p95; RPO 15 minutes, RTO 4 hours, because it can be rebuilt.

What it buys

• Off-road components appear in the estate picture, which is what makes ≥ 99% control coverage a measurable claim.
• The catalog's rebuildability lets it sit in the projection tier of the storage model with an RTO rather than RPO 0.
• Catalog accuracy becomes an SLO with a number, not an aspiration.

What it costs

• A standing divergence backlog that will never be empty and must not be allowed to become noise.
• Owner lookup during reconciliation is the one place the catalog sits in a critical path; it is cached and degrades to last-known-good.

Choose differently when. If ownership could be inferred reliably from runtime metadata alone — enforced at creation and never editable afterwards — discovery-only would become viable and the declaration path could shrink to intent. That is close to what ADR-11 aims at, and this decision would deserve revisiting once attribution is provably complete.

Why it holds up over time. Declared versus observed is a permanent tension in any inventory system. Naming the rule for who wins, per field rather than per system, is what keeps it from being relitigated every time the two disagree.

Lesson. An inventory that can only be right is an inventory that lies. Build the one that can say where it disagrees with reality.

ADR-10 · Ownership resolves to a live identity group, and entitlement derives from it

Status: Accepted · Shown on views: 12, 19, 21

How does the platform know who may act on a component, and what happens when that answer goes stale?

Context. The usual design keeps an access list beside the component: who can deploy, who can provision, who can read the secrets. It is correct on the day it is written and decays from then on, because teams reorganise and people move and nobody remembers the platform's list. Meanwhile the organisation already maintains an accurate, audited answer to the same question in its identity provider, updated by joiner-mover-leaver processes that exist whether the platform uses them or not.

Decision. A component's owning team resolves to an identity group with live membership. Entitlement to deploy, provision, read secrets and grant access is derived from that ownership rather than from a separate list. A component whose owning group is empty or deleted is an incident: it is quarantined, reported and reclaimed on a declared schedule rather than left running unattributed.

How it is realised on Google Cloud. team.entra_group_id is unique and required; the platform API resolves the caller's group claims against catalog ownership on every request. Orphan detection runs continuously. Bulk re-attribution is supported for reorganisations, and ownership history is retained.

What it buys

• A reorganisation moves entitlements with the components, in one operation, with history.
• There is no separate access review for the platform: reviewing group membership reviews platform access.
• Orphaned components are caught as a transition failure rather than discovered in a cost report.

What it costs

• A wrong owner in the catalog is now an access-control fault, not a metadata fault, which raises the stakes on descriptor quality.
• The platform inherits the identity provider's latency and availability on the authorisation path, so group claims are cached with a short lifetime.
• Quarantining an orphan is an unpopular act that must be preceded by notice and a grace period.

Choose differently when. In an organisation whose identity groups do not reflect engineering teams — where the directory models cost centres or legal entities instead — deriving entitlement from it would be worse than a maintained list. The precondition is explicit, and it is an assumption rather than a fact.

Why it holds up over time. Deriving authorisation from a single, independently maintained ownership fact is a pattern that survives any identity technology. The failure it prevents — a second list nobody updates — is as old as access control.

Lesson. Every access list you maintain is a list you will be wrong about. Derive authority from a fact somebody else is already obliged to keep true.

ADR-11 · Owner, cost centre, tier and classification are applied at creation as enforced properties

Status: Accepted · Shown on views: 14, 18, 19

When does a resource acquire the metadata that makes it attributable?

Context. Attribution added afterwards is never complete. A tagging campaign reaches the resources somebody remembered, an untagged resource has no owner to ask, and the gap grows faster than the campaign closes it. The FinOps reconciliation spreadsheet exists because attribution was optional at the moment of creation. The same is true of data classification, which decides permitted regions and storage, and of tier, which decides which guardrails and which SLO expectations apply.

Decision. Owner, cost centre, lifecycle state, tier and data classification are applied to every provisioned resource and every deployed workload at creation, as enforced properties rather than conventions. A workload without them is not admitted; a resource without them is not created. Cost is shown on the component page at the point of change, not only in a monthly report.

How it is realised on Google Cloud. Tags applied by the provisioning modules and required by binding admission policy, sourced from the catalog descriptor. Cost Management daily exports join directly to catalog ownership with no reconciliation step, because the join key was correct when the data was produced.

What it buys

• Cost attribution needs no reconciliation exercise, and idle-resource reclamation has an owner to notify.
• Data classification is available to policy at admission, so region and storage rules can be enforced rather than audited.
• The tier drives which guardrails apply, which is what makes an experimental sandbox defensible.

What it costs

• Every provisioning path must carry the metadata, including paths added later — a standing discipline on the platform team.
• A team that genuinely does not yet know its cost centre is blocked at creation, which needs a sane default rather than an exception.

Choose differently when. If the cloud provider offered reliable ownership inference from its own control-plane records, the enforcement could relax to verification. It does not, and inferred ownership is the specific failure mode this decision exists to prevent.

Why it holds up over time. Attribution at the point of creation is the only form of attribution that stays complete, on any platform and in any era. Retrofitting it has never worked anywhere.

Lesson. Metadata that is optional at creation is metadata you do not have. Make the thing impossible to create without it, or plan to spend years asking.

ADR-16 · Storage is classified by what a loss would cost, and only one band carries RPO 0

Status: Accepted · Shown on views: 11, 17, 20

Which of the platform's stores must never lose data, and which are allowed to be rebuilt?

Context. The default answer — back everything up, aim for zero loss everywhere — is expensive, and worse, it hides which stores actually matter. A platform holds several very different kinds of data: declarations it does not own, state it cannot reconstruct, projections it can rebuild in an afternoon, and working data whose loss nobody would notice. Treating them alike means the genuinely irreplaceable object gets the same attention as a pipeline log.

Decision. Four bands. Truth owned by teams lives in their repositories and is not the platform's to protect. Platform-owned truth — provisioning state, audit evidence, signed artefacts and policy bundles — carries RPO 0 with an exercised restore. Projections — catalog, dependency graph, insight stores — carry an RTO and are rebuildable from truth and the runtime. Working data is explicitly not backed up, and saying so is what stops it acquiring a policy nobody needs. One artefact, signed and attested, is promoted unchanged through every environment, so what was tested is what runs.

How it is realised on Google Cloud. Terraform state: RPO 0, RTO ≤ 1 hour, versioned and locked. Audit: RPO 0, immutable with legal hold, unwritable by platform operators. Catalog: RPO 15 minutes, RTO 4 hours. Portal: RPO 24 hours. Full control-plane rebuild from source and backup exercised at least twice a year.

What it buys

• The dangerous object is named: provisioning state, whose loss means the platform no longer knows what it created.
• The catalog's rebuildability is a designed property that keeps a large store out of the zero-loss tier.
• A restore that is exercised twice a year is a restore that works.

What it costs

• Rebuild paths for the projection tier must actually be maintained and tested, or the RTO is fiction.
• Immutable audit storage with legal hold is a cost that grows for seven years and cannot be trimmed retrospectively.

Choose differently when. If the catalog acquired data that exists nowhere else — a manually curated field, a human annotation — it would leave the projection tier that day. The classification has to be re-examined whenever a store gains a write that nothing else can reproduce.

Why it holds up over time. Classifying storage by reconstructability rather than by technology is a method, not a configuration. It applies unchanged to whatever stores the platform uses in ten years.

Lesson. Ask of every store: if this vanished tonight, what could we rebuild it from? The ones with no answer are the only ones that need zero-loss protection — and they are usually not the ones with the dashboards.

Identity, secrets and the control plane

Who may do what, and the fact that the platform is the most privileged thing in the estate.

ADR-08 · Exceptions are named, approved by someone else, expiring and published

Status: Accepted · Shown on views: 08, 15, 20

What happens when a binding guardrail blocks something the organisation has decided to do anyway?

Context. Relief always exists. If the architecture does not provide it, it appears as a quiet policy edit for the loudest team, as a permanently disabled check, or as a team routing around the platform entirely. Each of those is worse than an exception, and each is invisible. The question is therefore not whether to allow exceptions but what an exception has to cost in visibility.

Decision. A binding guardrail may be excepted for a named component, by an approver who is not the requester, with a stated reason and a mandatory expiry. Every active exception appears on a standing register that is reported alongside coverage. Expiry and a distinct approver are schema constraints, not process guidance. Authoring a policy, approving an exception to it, and deploying are three separate entitlements.

How it is realised on Google Cloud. exception carries approver_id (constrained to differ from the requester) and a non-null expires_at. Active exceptions are surfaced in the pull request of the component they cover, on the component page, and in the coverage report. Target: ≤ 25 active exceptions, each expiring inside 90 days.

What it buys

• The exception register becomes an honest measure of where policy and reality disagree.
• A control waived more often than it blocks is visible as a policy defect rather than a security success.
• Auditors get a complete answer: the control, its coverage, and every documented departure with an owner.

What it costs

• Expiry means recurring renewal work for genuinely long-lived exceptions, which teams experience as friction.
• A separate approver requires enough people with the entitlement that approval is not itself a queue.

Choose differently when. For a small number of controls — a hard regulatory floor, for example — no exception path is correct, and the architecture supports declaring a policy unexceptable. The general mechanism is still needed for everything else.

Why it holds up over time. Every enforcement system that lasts grows an exception path. Designing it deliberately, with expiry and attribution, is what separates a control that is trusted from one that is quietly disabled.

Lesson. If your control has no legitimate relief valve, you do not have a strong control — you have an unmeasured one, and someone has already routed around it.

ADR-15 · Workload identity everywhere, and the control plane is held to the controls it enforces

Status: Accepted · Shown on views: 20, 21

What credentials exist in this system, and who guards the thing that guards everything?

Context. The platform can provision, deploy and grant across the entire estate. It is, by construction, the highest-privilege system in the organisation, and it is usually the one exempted from its own controls because its team wrote them. Meanwhile static credentials accumulate wherever a pipeline needs to reach a cloud: in a secret store, in a variable, in a repository, in somebody's notes.

Decision. No long-lived static credential exists on any platform path. Workloads, pipeline runs and platform components authenticate by federated workload identity; a pipeline run's privileges are scoped to the component it is building. Secrets are brokered, never held: the platform creates a Key Vault path and binds an identity to it, and never reads secret material. The platform's own components are production systems under the same guardrails, the same change review and the same audit as tenants, and the entitlements to author policy, approve an exception and deploy are held separately.

How it is realised on Google Cloud. Entra workload identity federation for runners and workloads; ephemeral runners so build identity cannot outlive the build. Control-plane components deploy through the same reconciler and pass the same admission policy. Every privileged action — exception granted, policy changed, template published, ownership reassigned — is written to the append-only audit store.

What it buys

• Nothing to rotate, leak or discover in a repository.
• Compromising one component's build cannot deploy another team's component.
• The platform team experiences its own controls, which is the fastest feedback loop on whether they are reasonable.

What it costs

• Federation setup is per-workload work the platform must automate or it becomes the bottleneck it was built to remove.
• Holding the control plane to its own policy occasionally blocks the platform team during an incident, and the exception path (ADR-08) is the answer rather than a bypass.

Choose differently when. A legacy system that cannot federate needs a static credential with a short lifetime and a named owner; the architecture should carry that as an explicit, expiring exception rather than pretend the rule is universal.

Why it holds up over time. Short-lived, workload-scoped identity replaced static secrets everywhere it became available, and will not be reversed. Holding the control plane to its own rules is older still.

Lesson. The system that enforces the rules is the most valuable target in the estate. Exempting it from its own controls is how you find out.

The runtime and the fleet

The tenancy boundary, and recovery that does not depend on the machinery that broke.

ADR-13 · Rollback does not depend on the pipeline that produced the deploy

Status: Accepted · Shown on views: 06, 17

What is the recovery path when the thing that deployed the change is itself the problem?

Context. The convenient rollback is a pipeline run: re-deploy the previous version through the same machinery. It works for the common case and fails at the worst one, because the population of incidents where rollback is needed overlaps heavily with the population where the build plane, the registry pull, or the pipeline's own credentials are implicated. An engineer at 03:00 then has two problems, and the second one has a queue.

Decision. Rollback is a single, always-available operation that reverts the workload reconciler to the previously converged state, using an artefact already present, without requiring a build, a pipeline run or a registry round trip to succeed. Progressive delivery triggers it automatically on a declared signal breach, and a human can invoke it directly from the CLI.

How it is realised on Google Cloud. Argo CD holds the previous desired state; images for the previous revision are already pulled on the nodes or present in the geo-replicated registry. Target ≤ 5 minutes p99 from initiation to serving the previous version. Automated rollback fires on canary analysis breach without waiting for a human.

What it buys

• Recovery time is bounded by reconciliation, not by build time.
• The on-call journey in view 06 has an action that works without understanding the failure first.
• Automated rollback on a canary breach becomes credible, because the mechanism is simple enough to trust unattended.

What it costs

• The previous artefact must remain available, which constrains registry retention and node image garbage collection.
• A rollback that reverts intent can conflict with a fix already in flight, so the operation records itself in intent rather than mutating the cluster behind the reconciler's back.

Choose differently when. For a workload with irreversible data migrations, rollback is not a recovery strategy and the golden path for it must say so explicitly rather than offering a button that lies. That is a property of the path, not of the mechanism.

Why it holds up over time. Recovery machinery that shares fate with the failure is a recurring mistake rather than a technology-specific one. Independence of the recovery path outlives every deployment tool.

Lesson. Test your recovery path against the assumption that the thing you normally use is broken. That is the only case where it matters.

ADR-14 · Namespace per component in shared clusters, with staged fleet upgrades and a canary component per path

Status: Accepted · Shown on views: 16, 18

Where does the tenancy boundary sit, and who absorbs the cost of the answer?

Context. Cluster per team is strongly isolated and multiplies fleet operations, upgrade toil and idle capacity by 140 against a twelve-person platform team. Shared clusters are cheap to operate and make every noisy neighbour, every cluster-scoped resource and every upgrade a shared-fate event. The honest question is not which is safer in principle but which failure the organisation can actually staff.

Decision. Namespace per component in shared clusters, separated by environment and by subscription rather than by team, with experimental workloads in a sandbox subscription. Isolation inside a cluster is enforced rather than conventional: required requests and limits, per-namespace quota, default-deny network policy, workload identity per component. Upgrades roll through the fleet in stages with automated tenant smoke checks and abort-and-hold on failure, and each golden path has a canary component that walks the whole path continuously.

How it is realised on Google Cloud. Production and non-production in separate workload subscriptions; uk-south primary with uk-west paired for the platform's own recovery. Three admission webhook replicas per cluster with a locally cached bundle so enforcement survives a control-plane outage.

What it buys

• A fleet small enough that upgrades are a planned operation rather than a permanent campaign.
• Idle capacity is shared, which is most of the platform's cost advantage.
• Enforced limits and default-deny networking make the namespace boundary a control rather than a convention.

What it costs

• Cluster upgrades are shared-fate events and must be staged, smoke-checked and abortable.
• A cluster-scoped resource conflict between tenants is possible and has to be policed by policy.
• The highest-isolation workload in the estate sets the ceiling; if one arrives that the namespace cannot hold, the boundary has to move.

Choose differently when. A regulated workload with a hard physical or tenancy separation requirement flips this immediately for that workload — hence the deferred option rather than a rejected one. A platform team of 40 rather than 12 would also change the arithmetic.

Why it holds up over time. The boundary question — isolation against operability — recurs on every substrate. What this decision fixes is that the answer is chosen by what can be staffed, and that the alternative is designed for rather than argued away.

Lesson. Isolation you cannot operate is not isolation. Pick the boundary your team can actually maintain, and write down what would move it.

Platform as a product

The decisions that only make sense if teams are allowed to say no.

ADR-03 · Components track their golden-path template by default, and forking is a recorded, supported choice

Status: Accepted · Shown on views: 05, 07, 12, 19

After scaffolding, does the generated code belong to the team, to the platform, or to both?

Context. Scaffold-and-forget is the simplest contract and the most expensive one. Within a year the estate holds 1,400 divergent copies of a pipeline, and a security fix in the template reaches none of them. The opposite — the platform owning generated code in the team's repository — makes updates shippable and makes every product repository contain code its owners may not change, which is the fastest way to make a platform resented. The real variable is not who owns the file but whether the platform can tell the difference, and whether the team made the choice deliberately.

Decision. Every scaffolded component declares whether it tracks its template or has forked from it. Tracking components receive automated update pull requests; forked components do not, and their departure is recorded with a stated reason. Drift is measured per component, published to the owning team, and a supported floor is enforced with a published deprecation window. The shared pipeline is consumed by reference rather than copied, so the most change-prone artefact is not part of the fork surface at all.

How it is realised on Google Cloud. template_version and tracks_template are columns on the component record, populated from the repository descriptor. The update bot opens one pull request per tracking repository. The pipeline is a referenced reusable workflow; forking it is possible and is exactly the act that shows up in the drift report.

What it buys

• A template fix reaches most of the estate as automation rather than as a campaign.
• The fork ratio becomes a measured number that predicts migration cost — the single most useful planning figure the platform has.
• Leaving the paved road is legitimate and visible, which keeps adoption honest (ADR-12).

What it costs

• Automated pull requests into product repositories need organisational consent and careful batching, or they become noise teams filter out.
• The forked tail absorbs disproportionate platform-team effort — an assumed 16% of components in view 05.
• Two supported template versions at a time is the practical ceiling for a twelve-person team; more is a commitment that will be broken.

Choose differently when. If the organisation adopted a mandate — every component must be on a current template — tracking could be assumed and the fork machinery would be waste. If automated pull requests proved culturally unacceptable, the fallback is a stronger by-reference boundary: move more of the template into consumed artefacts and generate less.

Why it holds up over time. The distinction between generated-and-owned and consumed-by-reference is older than any of this tooling and will outlive it. What changes is where the line sits, and the architecture is built so that line can move without redesign.

Lesson. Generated code is a fork with good manners. Plan for the divergence at the moment you generate it, because the alternative is discovering the divergence when you need it not to exist.

ADR-07 · A thin abstraction with strong defaults, and an escape hatch that is a designed path

Status: Accepted · Shown on views: 03, 06, 16

How much of the underlying runtime should the golden path hide from the engineer who uses it?

Context. The highest abstraction gives the best first day: declare a service, get a service, never learn what a pod is. It leaks at the first incident, which is exactly when the person holding the pager has the least capacity to learn a substrate the platform has spent a year hiding. A thin abstraction costs every engineer the learning and traps nobody. The choice also decides whether the platform can change the substrate later without its tenants noticing — and who is expected to do the migration when it does.

Decision. The golden path presents a thin abstraction with strong, cost-aware defaults: the underlying runtime remains visible and addressable, and the platform supplies the configuration rather than concealing the model. Read access to the substrate for a component's owners is an entitlement-scoped, audited path, not an exception granted during an incident. What the platform hides is toil, not the model.

How it is realised on Google Cloud. Manifests are real and readable in the team's repository. The CLI grants scoped cluster read access derived from catalog ownership, with every access audited. The component page links directly to the runtime objects rather than paraphrasing them.

What it buys

• The on-call journey in view 06 is survivable without a platform engineer.
• Escape-hatch use is measurable, and a spike in it names the abstraction that is failing.
• Teams can reason about cost and behaviour because they can see what actually runs.

What it costs

• Every engineer carries more concepts than the highest-abstraction alternative would require.
• Substrate migrations are visible to tenants and must be negotiated rather than performed silently.
• The cognitive-load budget has to be stated and defended, because each golden path will want to add one more concept.

Choose differently when. With a junior-heavy engineering population, a dedicated platform on-call carrying tenant incidents, or a substrate the platform genuinely intends to replace, a higher abstraction becomes the better trade — and the escape hatch becomes the platform team's tool rather than the tenant's. This is the decision most likely to be revisited, and the requirement leaves it explicitly open.

Why it holds up over time. The tension between leverage and comprehensibility does not age. What changes is where the line sits; keeping the substrate addressable means the line can be moved later in either direction without a rewrite.

Lesson. An abstraction is a promise about the good days and a debt on the bad ones. Price it at 03:00, not at the demo.

ADR-12 · The API is primary, and no platform component sits in the critical path of running production

Status: Accepted · Shown on views: 09, 16, 18

What may fail when the platform fails?

Context. A platform that is loved on good days and catastrophic on bad ones does not survive its first outage. The temptation is to make the portal the centre: it holds the session, it brokers the deploy, it proxies the runtime. Every one of those choices puts a convenience component into a path that must stay up when the platform is down. Separately, a capability that exists only in the portal cannot be scripted by the teams that most need it, cannot be tested, and cannot be replaced when the portal is.

Decision. The platform API is the primary surface; the portal and the CLI are clients of it, and a capability reachable only through one of them is a defect. No platform component is a dependency of serving production traffic: deploys, rollbacks and reconciliation continue when the portal is unavailable, and running workloads are never evicted because the platform cannot evaluate them (ADR-06). The platform publishes its own SLOs and error budget to its tenants.

How it is realised on Google Cloud. Portal and catalog read path ≥ 99.5% monthly; admission evaluation ≥ 99.95%; the portal carries RPO 24 hours because it holds no truth. The API is versioned with a published deprecation window, since its consumers include 140 teams' pipelines.

What it buys

• A portal outage is an inconvenience with a known blast radius, and can be stated as such in advance.
• Every capability is scriptable, which is what lets teams automate around the platform rather than through it.
• The platform can replace its own portal without renegotiating with its tenants.

What it costs

• Two clients to keep at parity, and a standing temptation to add portal-only behaviour under deadline.
• A versioned API with a deprecation window is a commitment that constrains the platform's own pace of change.

Choose differently when. If the platform ever needed to sit in a request path — a runtime configuration lookup, say — that component would have to be designed to the availability of the thing it serves, and this decision would be restated for it rather than quietly broken.

Why it holds up over time. Keeping the control plane out of the data path is one of the few architectural rules that has never stopped being right. It survives every technology change because it is a statement about blast radius.

Lesson. Decide in advance what your outage is allowed to cost. If the answer includes other people's production traffic, you have built a dependency rather than a platform.

Every package used, in one table

Eleven terms that carry specific meaning in this package. Several are used loosely in the wider platform-engineering literature; here they are the names of decisions.