# Architecture Decision Record

*Service Mesh Platform · Solution Architecture v1.0 · Platform Networking Architecture · 2026-09 · 31 views · open source, fully on-premises*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Thirty-two decisions make up this architecture. Everything else on the thirty-one views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the proxy, the mesh project or the team has changed. Read the one-pager first. Read a record when you want to argue with it.

> **Status of this document.** This is a design, not a report on a running system. Latencies, resource figures and rates are the requirement's targets or planning estimates, and each is replaced by a measurement in the proof phase. Seven product behaviours are relied on and must be confirmed on the exact releases chosen before production depends on them: Envoy's TLS stack enforcing URI name constraints on a per-cluster intermediate; Istio consuming SPIRE-issued SVIDs over the SPIFFE Workload API with Kubernetes native sidecars; a SPIRE server on CloudNativePG signing 2,000 SVIDs a second; Envoy's retry budget and expected-timeout handling applied through one EnvoyFilter that survives two consecutive Istio minor upgrades; Istio's dry-run mode for authorisation policies, which upstream still marks experimental, behaving as documented at 2,000 proxies per cluster; idle CPU and scoped memory per sidecar at 2,000 services; and OpenBao unsealing from a hardware-backed key on the release deployed.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it works on-premise:** The concrete mechanism: which package, configured how, on whose hardware.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**Mesh shape**: What is on the request path, which mesh, where the proxy lives, what a proxy does alone, and how the control plane is divided.

- ADR-01 · The control plane is a configuration compiler, never a request-path dependency
- ADR-02 · Istio with Envoy as the mesh, standard APIs preferred over Istio-specific ones
- ADR-03 · A sidecar per workload now; a per-node L4 layer later, only where no L7 is needed
- ADR-04 · Proxies serve last-known-good configuration indefinitely; certificate lifetime is the only bound
- ADR-05 · One control plane per cluster, sharded by cluster, with no cross-site dependency
- ADR-06 · Native sidecars for lifecycle ordering; injection failure behaviour declared per namespace

**Identity and trust**: Where a workload's name comes from, how long it lasts, and how one root becomes twenty clusters that cannot vouch for each other.

- ADR-07 · SPIRE issues workload identity from node and workload attestation
- ADR-08 · 24-hour workload certificates renewed at 12 hours; short lifetime replaces revocation lists
- ADR-09 · Offline HSM root, name-constrained per-cluster intermediates in OpenBao, roots-only trust bundle

**Transport and authorisation**: How a fleet moves from plaintext to strict, where the allow-or-deny decision is made, how a policy is proven before it bites, and what leaves the mesh.

- ADR-10 · Move from permissive to strict per service, measured, by reviewed change
- ADR-11 · Authorisation is compiled into the destination proxy; external authorisation is a per-service opt-in
- ADR-12 · Default deny in strict mode, dry-run before enforcement, and an impact gate at merge
- ADR-13 · Egress only to declared hosts through an egress gateway, with Cilium as the second enforcement plane

**Traffic and resilience**: How routes are written, who owns a weight, how fast a mistake is undone, and how retries and overload are kept from becoming outages.

- ADR-14 · Routing intent is written in the Kubernetes Gateway API; Istio APIs only where the standard has no field
- ADR-15 · The rollout controller owns traffic weights during a rollout, and judges the share the mesh reports
- ADR-16 · Revert pins a previously rendered bundle, and is the one operation allowed ahead of Git
- ADR-17 · Timeouts are mandatory, retries are budgeted, and deadlines travel with the call
- ADR-18 · Outlier ejection and load shedding at the proxy; locality routing with a declared spillover threshold

**Multi-cluster and network**: How traffic crosses a cluster or a site, how trust between clusters is declared, and what the network under the mesh provides.

- ADR-19 · Flat pod networking inside a site; east-west gateways between sites
- ADR-20 · Cross-cluster trust and service export are declared, never default
- ADR-21 · Cilium for pod networking, BGP-announced gateway addresses and a second enforcement plane

**Change and governance**: Where intent lives, who may change which part, how a change and an upgrade reach 40,000 proxies, and what is recorded.

- ADR-22 · Intent in three repositories, with separation of duties enforced by repository and by Argo CD project
- ADR-23 · Argo CD per site, and every mesh change rolls out in cluster waves behind health gates
- ADR-24 · Proxy configuration is scoped to declared dependencies and distributed as deltas
- ADR-25 · Admission guards are the second line: OPA Gatekeeper and ValidatingAdmissionPolicy
- ADR-26 · Proxy fleet upgrades by control-plane revision, never in place
- ADR-27 · An append-only audit trail on Object Lock storage, independent of the SIEM

**Observability and cost**: What every hop reports, how a failure is attributed, how drift is seen, and how the mesh's own cost is kept inside a number.

- ADR-28 · Prometheus per cluster with Thanos for global query and history; labels pruned at the proxy
- ADR-29 · Access logs to ClickHouse and traces to Tempo, sampled per namespace, never blocking a request
- ADR-30 · Every failure signal carries its origin: application, mesh policy or configuration, or network
- ADR-31 · Drift is observed per proxy, not inferred from sync status
- ADR-32 · The mesh's cost is a declared budget, measured monthly and attributed to teams with OpenCost

## Technology by capability

Every capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns; no managed service and no external dependency sits on any request path.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Request-path rule | Compile ahead, enforce locally | This design | Central decision per request | Control-plane outage becomes a freeze, not an outage | ADR-01 |
| Service mesh | Istio (sidecar mode) | Open source | Linkerd; Cilium service mesh | Full L7 feature set, multi-cluster, revisions, per-node mode in the same project | ADR-02 |
| Data-plane proxy | Envoy as a native sidecar | Open source | Per-node L4 proxy (ztunnel) with waypoints | Blast radius of one pod while the platform is young | ADR-03 |
| Behaviour with control plane down | Last-known-good, bounded by certificate lifetime | This design | Staleness TTL then fail closed | One clock bounds survival, not two | ADR-04 |
| Control-plane topology | istiod per cluster, multi-primary | Open source | Primary per site with remote clusters | Shards at about 2,000 proxies and contains failure per cluster | ADR-05 |
| Proxy lifecycle and injection | Kubernetes native sidecars; ValidatingAdmissionPolicy backstop | Open source | Classic sidecar with startup hold | Ordering by the kubelet; strict namespaces never admit unmeshed pods | ADR-06 |
| Workload identity issuance | SPIRE with k8s_psat and kubelet attestation | Open source | istiod CA; cert-manager istio-csr | Attested identity and issuance evidence, no key in istiod | ADR-07 |
| Certificate lifetime | 24 h SVIDs, renew at 12 h, no CRL | This design | 1 h certificates; 30 d with OCSP | Gives the 12-hour outage budget and a one-day exposure | ADR-08 |
| Clock discipline | chrony and node-problem-detector taint at ±60 s | Open source | PTP everywhere | 60 s is ample against a 24-hour certificate | ADR-08 |
| Root and intermediates | Offline HSM root; name-constrained intermediates in OpenBao | Open source | Per-cluster roots with federation | One root, cryptographic containment per cluster | ADR-09 |
| mTLS migration | Per-service PeerAuthentication after 7 days of zero plaintext | This design | Mesh-wide strict on a date | Each flip backed by evidence about real callers | ADR-10 |
| Authorisation | Istio AuthorizationPolicy in Envoy RBAC | Open source | External authoriser on every call | 0.2 ms p99 and no network dependency | ADR-11 |
| External authorisation (opt-in) | Envoy ext_authz to an OPA sidecar | Open source | Central policy service | Live context only where declared, with a pod-local failure domain | ADR-11 |
| Policy proof | Dry-run annotation plus access-log replay gate at 0.5% | This design | Enforce and roll back on complaints | Proven against yesterday's and today's traffic before it can deny | ADR-12 |
| Egress control | ServiceEntry, egress gateway, Cilium egress policy | Open source | Perimeter firewall only | Attributable external calls; bypass-resistant | ADR-13 |
| Routing API | Kubernetes Gateway API (mesh profile) | Open source | Istio VirtualService throughout | Portable intent for what teams edit most | ADR-14 |
| Progressive delivery | Argo Rollouts with analysis on Thanos | Open source | CD pipeline writing weights | One writer of weights; judgement on applied share | ADR-15 |
| One-step revert | Signed OCI bundles in Harbor, pinned by a revert controller | This design | Git revert and sync | Undo does not wait on the pipeline that made the mistake | ADR-16 |
| Timeouts, retries, deadlines | Mandatory route timeouts, Envoy retry budget 20%, expected-timeout header | Open source | Attempt counts only | Budgets bound the fleet, not just one call | ADR-17 |
| Overload and locality | Outlier ejection, circuit limits, locality failover at 70% | Open source | Client-side libraries | Shed at the proxy; spillover is declared, not implicit | ADR-18 |
| Cross-cluster transport | Flat inside a site; SNI east-west gateways between sites | Open source | Flat network everywhere | Shortest path where the network is ours; containment where it is not | ADR-19 |
| Federation | Exported services and trusted domains declared per cluster | This design | Merge all discovery by default | Trust and discovery are opt-in and reviewable | ADR-20 |
| Pod network and load balancing | Cilium with BGP control plane and LB IPAM | Open source | Calico with MetalLB | One component for CNI, VIPs and the second enforcement plane | ADR-21 |
| Intent store | Forgejo, three repositories, signed commits | Open source | GitLab CE; one monorepo | Separation of duties by repository, not by review convention | ADR-22 |
| Configuration delivery | Argo CD per site, cluster waves with health gates | Open source | Flux; a central Argo CD | No cross-site dependency; waves align with control-plane blast radius | ADR-23 |
| Configuration scope | Istio Sidecar resources generated from declared dependencies; delta xDS | Open source | Inferred dependency graph | Memory tracks what a workload calls; changes cost what they touch | ADR-24 |
| Admission | OPA Gatekeeper and ValidatingAdmissionPolicy | Open source | CI checks only | A second line that kubectl cannot skip | ADR-25 |
| Mesh upgrades | Istio revisions and revision tags, namespace waves | Open source | In-place upgrade | Skew held by construction; abort leaves a working mixed fleet | ADR-26 |
| Audit trail | Git history, Kubernetes audit, SPIRE audit to Ceph Object Lock | Open source | SIEM only | Append-only for 13 months, independent of the SIEM's retention | ADR-27 |
| Metrics | Prometheus per cluster with Thanos on Ceph RGW | Open source | VictoriaMetrics; Grafana Mimir | Cluster-local collection, global query, 13-month downsampled history | ADR-28 |
| Access logs and traces | OpenTelemetry Collector to ClickHouse and Grafana Tempo | Open source | OpenSearch; Jaeger | Columnar logs at 800k hops a second; object-storage traces | ADR-29 |
| Failure attribution | Failure-origin label from Envoy response flags | This design | Status codes only | Separates the mesh's failures from the application's | ADR-30 |
| Drift visibility | Drift exporter: acknowledged sha and nonce per proxy | This design | Argo CD sync status; istioctl on demand | Effective state is observed, not inferred | ADR-31 |
| Cost attribution | OpenCost plus telemetry volume per namespace | Open source | Spreadsheet estimates | The team that meshes a service sees what it costs | ADR-32 |

## The decisions, and the alternatives that lost

### Mesh shape

*What is on the request path, which mesh, where the proxy lives, what a proxy does alone, and how the control plane is divided.*

#### ADR-01 · The control plane is a configuration compiler, never a request-path dependency

**Status:** Accepted  ·  **Shown on views:** 02, 07, 11, 21

*What is a proxy allowed to consult while it forwards a request?*

**Context.** A mesh puts a proxy on every request the organisation serves. Whatever that proxy depends on at request time inherits the availability target of every service in the fleet: 99.99% for the mesh-attributable data path. The control plane, the CA and the configuration store are complex, change weekly and target 99.9%. If any of them is consulted per request, even as a cache-miss fallback, the fleet's availability becomes theirs.

**Decision.** A proxy forwards a request using only three things it already holds: the last configuration it acknowledged, the trust bundle, and an unexpired certificate for its own identity. istiod compiles intent into per-proxy configuration ahead of time and streams it; SPIRE issues certificates ahead of expiry; neither is called while a request is in flight. When they are down, traffic continues on the last good state and new intent waits.

**How it works on-premise.** Envoy receives listeners, routes, clusters, endpoints and RBAC rules over xDS from istiod and holds them in memory. Certificates and the bundle arrive over SDS from the node's SPIRE agent, which caches them. External authorisation filters are not configured unless a service opts in (ADR-11). A conformance test in the platform pipeline blocks any EnvoyFilter or mesh setting that adds a synchronous call to a control-plane component.

| Option | Verdict | Reasoning |
|---|---|---|
| Compile ahead, enforce locally | Chosen | Control-plane outages become freezes. The mesh's added latency is a property of the proxy alone and can be budgeted in microseconds. |
| Central policy or routing decision per request | Rejected | Can use live state and propagates changes instantly. Puts a network call and its tail latency into every request, and makes its availability the fleet's. |
| Local by default, fetch on cache miss | Rejected | Looks like the best of both. The miss path is exercised only during incidents, which is when the remote service is most likely to be unavailable. |
| Library in every service instead of a proxy | Right elsewhere | Right for a single-language estate that can redeploy every service for a policy change. Here there are many languages and 250 teams. |

**What it buys**

- A total control-plane outage costs new changes and new workloads, not existing traffic
- The per-hop latency budget depends only on proxy behaviour, which is measurable in isolation
- Control-plane components can be upgraded and restarted without a change window for services

**What it costs**

- Effective state can diverge from intent for as long as distribution is broken; a revoked permission is not revoked until the push lands
- Every policy change is a configuration distribution, with its latency and its failure modes
- Anything that needs per-request live state (a user's entitlements, a risk score) cannot be expressed as mesh policy

**Choose differently when.** When authorisation must reflect state that changes per request and cannot be compiled, such as per-user entitlements, put that decision in an external authoriser for the services that need it and accept its latency and failure mode explicitly, without moving the rest of the mesh onto that path.

**Why it holds up over time.** This is a rule about dependency direction, and it does not name a product. A different proxy, a different mesh or a per-node data plane all fit behind it unchanged, and any future design review can test a proposal against one question: does this add something the proxy must ask during a request?

> **Lesson.** Put what decides on a different availability budget from what serves, and make the serving side able to run on what it already holds.

#### ADR-02 · Istio with Envoy as the mesh, standard APIs preferred over Istio-specific ones

**Status:** Accepted  ·  **Shown on views:** 02, 07, 08

*Which open-source mesh carries the requirement for the next decade?*

**Context.** The requirement needs L7 routing with 1% weights, header matching and mirroring, destination-side authorisation on identity with dry-run, outlier ejection, locality failover, cross-cluster federation through gateways that hold no identity, controlled egress and revision-based upgrades. It must be open source, run on-premise, and be supportable by a platform team for years without a vendor subscription.

**Decision.** Use Istio, a CNCF graduated project, in sidecar mode with Envoy as the proxy and SPIRE as the certificate source (ADR-07). Write routes in the Kubernetes Gateway API where it has the field and use Istio's own APIs only where it does not (ADR-14). Keep Istio's per-node L4 mode (ambient) as the Phase 3 option for workloads with no L7 needs (ADR-03).

**How it works on-premise.** Istio is installed per cluster with Helm charts pinned by revision, from images mirrored into Harbor and signed with cosign. The platform team builds nothing from source; it tracks upstream minor releases within the support window and applies patch releases within two weeks of publication.

| Option | Verdict | Reasoning |
|---|---|---|
| Istio with Envoy sidecars | Chosen | Broadest L7 feature set, multi-cluster with east-west gateways, revision upgrades, dry-run policy, SPIRE integration, and a per-node mode in the same project. |
| Linkerd | Rejected | Smaller, simpler, and its Rust proxy is lighter than Envoy. Since 2024 stable release artifacts are published only by the vendor, and weighted routing, mirroring and egress control are thinner than the requirement needs. |
| Cilium service mesh | Rejected | Already our CNI, and L4 is excellent. L7 runs in a per-node Envoy shared by many identities, and its mutual authentication does not bind each connection to a workload certificate the way mTLS does. |
| Consul | Rejected | Mature and works outside Kubernetes. Relicensed under the Business Source License in 2023, so it fails the open-source constraint. |
| Kuma | Deferred | Envoy-based and multi-zone by design. Smaller community and CNCF sandbox maturity; worth re-evaluating if Istio's direction changes. |

**What it buys**

- Every capability in the MVP and Phase 2 exists upstream; the platform team configures rather than builds
- Envoy is the most widely deployed L7 proxy, so its behaviour under failure is well documented by others
- Ambient mode gives a migration path to lower per-pod cost without changing mesh

**What it costs**

- Istio's configuration surface is large and easy to misuse; admission guards and a narrow set of allowed fields are needed (ADR-25)
- Envoy's memory floor per sidecar is higher than Linkerd's proxy
- Two features the requirement needs (retry budgets, deadline handling) are reachable only through EnvoyFilter today

**Choose differently when.** A single-cluster estate that needs mTLS, retries and golden metrics but no weighted routing, mirroring or controlled egress, and is willing to buy vendor support, gets most of the value from Linkerd at a lower resource cost and operational load.

**Why it holds up over time.** The choice is hedged at every seam: identity is SPIFFE and comes from SPIRE, routes are Gateway API objects, telemetry is OTLP. If Istio loses momentum, the migration is to another mesh that reads the same objects and the same certificates, not a rewrite of what 250 teams have declared.

> **Lesson.** Pick the product for today's requirement, and write your intent in the standards it implements, so that picking again later is a migration and not a rebuild.

#### ADR-03 · A sidecar per workload now; a per-node L4 layer later, only where no L7 is needed

**Status:** Accepted  ·  **Shown on views:** 08, 26, 27

*Should the proxy be one per workload or shared per node?*

**Context.** A sidecar gives per-workload identity, per-workload configuration scope and a blast radius of one pod. It costs a proxy's memory and idle CPU floor multiplied by 40,000 pods, which planning puts at about 400 vCPU idle and up to 4.8 TB of memory at the p95 target. A shared per-node proxy amortises that cost but multiplexes many identities in one process and fails a whole node at a time. A split, with L4 identity and encryption per node plus L7 per service only where needed, buys most of the saving for a second data-plane component.

**Decision.** Run Envoy sidecars for every meshed workload in the MVP and Phase 2. Every MVP capability, from subset routing to destination authorisation on path, is L7. In Phase 3, offer Istio's ambient mode (a per-node L4 proxy, ztunnel, with per-namespace L7 waypoint proxies) to namespaces that declare no L7 policy, once the proof phase has measured the saving and the node-level failure mode has been drilled.

**How it works on-premise.** Sidecars are injected as Kubernetes native sidecars. Proxy concurrency is set from the pod's CPU request, not left at the node's core count. The Phase 3 pilot moves batch and internal-tooling namespaces first, where a node-wide proxy failure is cheapest; the attribution in ADR-32 compares their cost per request before and after.

| Option | Verdict | Reasoning |
|---|---|---|
| Sidecar everywhere, per-node L4 opt-in later | Chosen | Smallest blast radius and simplest identity model while the platform is new; the cheaper model arrives when it can be measured. |
| Per-node L4 plus waypoints from day one | Deferred | Lower cost at scale. Adds a second data-plane component and a node-wide failure mode before the team has operated either. |
| Shared per-node L7 proxy | Rejected | Best resource amortisation. One process holds many workloads' keys, which contradicts the requirement that a compromised proxy can impersonate only its own workload. |
| Sidecar only for services that opt in | Rejected | Cheap, and leaves mTLS and identity optional exactly where teams have not thought about them. |

**What it buys**

- A proxy crash or compromise affects one pod and one identity
- Configuration can be scoped per workload, which keeps each proxy's memory tied to what it calls
- One data-plane model to operate, debug and upgrade in the first two phases

**What it costs**

- Idle floor multiplied by pod count is the largest single line in the mesh budget
- Pod startup includes proxy readiness, adding roughly a second to every pod start
- Two data-plane models coexist from Phase 3, with different debugging tools

**Choose differently when.** When pod density is high (dozens of small pods per node), most traffic needs only mTLS and L4 policy, and the organisation has run node-level agents like a CNI at scale without incident, start with the per-node model and add L7 proxies per service.

**Why it holds up over time.** The decision is expressed per namespace, not per fleet. As per-node data planes mature, namespaces move one at a time on measured cost, and nothing else in the architecture changes: identities are still SPIFFE, policies are still authored the same way, and telemetry keeps the same labels.

> **Lesson.** Buy the simplest failure model first and the cheapest resource model second, and make the switch a per-tenant setting rather than a platform migration.

#### ADR-04 · Proxies serve last-known-good configuration indefinitely; certificate lifetime is the only bound

**Status:** Accepted  ·  **Shown on views:** 21, 31, 06

*How should a proxy behave when the control plane is gone?*

**Context.** The options are to serve last-known-good forever, to serve it until a staleness timer then fail closed, or to keep existing connections but refuse new ones. Serving forever maximises availability and lets the fleet diverge silently from intent, including a revoked permission that never takes effect. A timer bounds divergence but turns a long control-plane outage into a data-path outage. Certificate lifetime already bounds how long a proxy can run alone.

**Decision.** Proxies keep their last acknowledged configuration with no staleness timer. The 24-hour certificate is the bound: after 12 hours without issuance, renewals fail and pages fire; after 24, pods fail closed one by one. The mesh promises that a deny propagates within 15 s p99 while the control plane is up and makes no promise while it is down. Emergency blocks during an outage use Cilium network policy, which does not depend on istiod.

**How it works on-premise.** No xDS staleness setting is configured on Envoy. The drift exporter reports xDS age per proxy, and a page fires when istiod has been unreachable for 5 minutes in any cluster. A crashed Envoy restarts under its container's restart policy and waits for istiod, while the pod stays unready and is removed from endpoints. The emergency-block runbook applies a CiliumNetworkPolicy selecting the pod by label, through the Kubernetes API, which is a separate failure domain from istiod.

| Option | Verdict | Reasoning |
|---|---|---|
| Serve indefinitely, bounded by certificate lifetime | Chosen | No second timer that could expire first; availability during an outage is as long as identity allows. |
| Staleness TTL, then fail closed | Rejected | Bounds divergence. A TTL shorter than the certificate converts a control-plane outage into a fleet outage; a longer one does nothing. |
| Persist xDS to local disk so crashed proxies restart without istiod | Rejected | Meets the requirement's proxy-crash row literally. Requires forking or extending the proxy agent and carrying that patch through every upgrade. |
| Refuse new connections when stale | Rejected | Protects against stale policy on new callers, and breaks every connection pool that recycles during the outage. |

**What it buys**

- A 12-hour total control-plane outage causes no request failures
- One number, the certificate lifetime, governs how long the fleet can run alone
- The deny-propagation promise is explicit, so security incident runbooks do not assume a push lands during an outage

**What it costs**

- A permission revoked during an outage keeps working until the control plane returns or the certificate expires
- Pods whose sidecar crashes during a total outage stay out of service until istiod returns; this is a recorded deviation from the requirement
- An emergency block during an outage is coarse (L3/L4 by pod label) rather than a precise L7 deny

**Choose differently when.** Where a revoked permission taking effect matters more than availability (a payment authorisation boundary under regulatory control, for example), put that boundary behind an external authoriser with fail-closed posture, or shorten certificates for those workloads, and accept the outage that follows.

**Why it holds up over time.** The rule does not depend on how configuration is delivered or how long certificates last; it depends only on there being one bound instead of two. If certificate lifetimes change, the survivable outage changes with them in the same record, and nothing else needs retuning.

> **Lesson.** Do not add a timer that fails before the one you already have. Decide which clock bounds survival, and state what the system will not promise while that clock runs.

#### ADR-05 · One control plane per cluster, sharded by cluster, with no cross-site dependency

**Status:** Accepted  ·  **Shown on views:** 08, 22, 23

*How is the control plane divided so that no replica holds the whole fleet and no region depends on another?*

**Context.** At the design ceiling, 40,000 proxies hold long-lived xDS streams. A single mesh-wide control plane would make one deployment the blast radius for every cluster and put cross-site latency into every configuration push. The requirement asks for multiple replicas per cluster, horizontal distribution, and no cross-region dependency for in-region configuration.

**Decision.** Each of the 20 clusters runs its own istiod (three replicas across three rooms) and its own SPIRE server set. Clusters are peers in an Istio multi-primary topology: each reads its own API server for local intent and endpoints, and reads peer clusters only for services those peers export (ADR-20). A cluster's control plane serves at most that cluster's proxies, about 2,000.

**How it works on-premise.** istiod runs on the cluster's platform node pool with a PodDisruptionBudget of one and a HorizontalPodAutoscaler on xDS connection count. Proxies connect through a cluster-local Service, so load spreads across replicas and a replica loss moves only its connections. Remote-cluster access uses a read-only Kubernetes service account per peer.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-cluster control plane, multi-primary | Chosen | Blast radius of one cluster, natural sharding at about 2,000 proxies, and no site depends on another for its own configuration. |
| One primary control plane per site, remote clusters | Rejected | Fewer istiod deployments. A site's primary becomes the blast radius for up to seven clusters and a single upgrade event for all of them. |
| One global control plane | Rejected | Simplest to reason about. Cross-site latency on every push and a fleet-wide blast radius. |

**What it buys**

- A bad istiod release or a runaway push is contained to one cluster, which is also the unit of rollout waves
- Each site keeps changing and serving during a site partition
- Control-plane capacity grows linearly with clusters, never with the fleet as a whole

**What it costs**

- Twenty control planes to upgrade, monitor and keep on compatible versions
- Remote-cluster discovery means each istiod watches peer API servers, which is load and credentials to manage
- Consistency across clusters is eventual; a change lands cluster by cluster by design

**Choose differently when.** With three or four clusters in one site on one network, a single primary control plane with remote clusters is simpler to run, and the blast-radius argument is weak because the clusters already share failure domains.

**Why it holds up over time.** Sharding by the unit that already owns failure (the cluster) means growth adds clusters rather than making any one component bigger. The design ceiling can be passed by adding a site without revisiting the control-plane topology.

> **Lesson.** Shard the control plane along the boundary that already contains failures, so one partitioning serves scale, blast radius and rollout at once.

#### ADR-06 · Native sidecars for lifecycle ordering; injection failure behaviour declared per namespace

**Status:** Accepted  ·  **Shown on views:** 08, 31, 04

*How does a proxy start before its application and stop after it, and what happens when injection is unavailable?*

**Context.** Classic sidecars race the application: an app that sends a request before Envoy is ready fails, and a proxy that exits before in-flight requests finish drops them. Injection is done by a mutating webhook served by istiod, so istiod being down means new pods either start unmeshed or do not start at all. The requirement forbids admitting a pod silently unmeshed into a strict namespace, and forbids a global default for the behaviour.

**Decision.** Inject Envoy as a Kubernetes native sidecar (an init container with restartPolicy Always), so the kubelet starts it and waits for its readiness before starting the application, and terminates it only after the application containers exit. Declare injection failure per namespace: strict namespaces fail closed, permissive namespaces fail open. An in-apiserver admission policy, independent of the webhook, refuses any pod without a sidecar in a strict namespace unless it carries a reviewed opt-out.

**How it works on-premise.** Two webhook configurations are registered with namespace selectors: one with failurePolicy Fail for namespaces labelled mesh.corp/mode=strict, one with Ignore for permissive ones. A ValidatingAdmissionPolicy (CEL, evaluated inside the API server, so it has no webhook to fail) checks for the istio-proxy init container in strict namespaces. Opt-outs are an annotation that admission accepts only when the pod's namespace lists that workload in the security repository.

| Option | Verdict | Reasoning |
|---|---|---|
| Native sidecars plus per-namespace failure policy and an independent admission check | Chosen | Lifecycle ordering done by the kubelet; injection outages degrade differently where the risk differs. |
| Classic sidecar with holdApplicationUntilProxyStarts and preStop sleeps | Rejected | Works on older Kubernetes. Relies on timing hacks that fail under slow nodes and long drains. |
| One global webhook failure policy | Rejected | Simple. Either every namespace stops scheduling when istiod is down, or strict namespaces admit unmeshed pods. |
| CNI-based injection with no webhook | Deferred | Removes the webhook dependency. Ambient mode does this for L4; it arrives with ADR-03's Phase 3 pilot. |

**What it buys**

- No startup or shutdown race between application and proxy
- A strict namespace cannot receive an unmeshed pod even if the webhook is misconfigured
- Permissive namespaces keep scheduling during an istiod outage

**What it costs**

- Requires Kubernetes with native sidecars (stable since 1.33) on every cluster
- Strict namespaces cannot scale up during an istiod outage in that cluster
- Opt-outs are a list the security team must keep short and current

**Choose differently when.** On a Kubernetes version without native sidecars, use the classic ordering settings and a longer termination grace period, and plan the cluster upgrade as a prerequisite for strict mode rather than as a nice-to-have.

**Why it holds up over time.** Lifecycle ordering has moved from mesh-specific workarounds into Kubernetes itself, which is where it will stay. The failure-policy split is expressed as namespace labels and an admission policy in core Kubernetes, and would apply unchanged to any injector.

> **Lesson.** Let the platform that owns process lifecycle own proxy ordering, and put the safety check for your most important invariant somewhere that cannot fail with the thing it checks.

### Identity and trust

*Where a workload's name comes from, how long it lasts, and how one root becomes twenty clusters that cannot vouch for each other.*

#### ADR-07 · SPIRE issues workload identity from node and workload attestation

**Status:** Accepted  ·  **Shown on views:** 09, 16, 12

*Where does a workload's cryptographic name come from?*

**Context.** The requirement insists that identity be derived from an attestation of the platform, never from a value the workload asserts, and that every issuance be recorded with the attestation that justified it. Istio's built-in CA authenticates a certificate request with the pod's projected service-account token, which is platform-issued but does not bind the request to an attested node, and its issuance logging is not designed as audit evidence.

**Decision.** Run a SPIRE server set per cluster and a SPIRE agent on every node. The agent attests its node with a projected service-account token checked against the API server; the agent then attests each workload by asking the kubelet which pod owns the requesting process. SVIDs are issued for identities of the form spiffe://<cluster>.mesh.corp/ns/<namespace>/sa/<service-account>. Envoy obtains its SVID and trust bundle over SDS from the SPIFFE Workload API socket. Registration entries are generated from Kubernetes objects, not written by hand.

**How it works on-premise.** SPIRE server uses the k8s_psat node attestor, a CloudNativePG PostgreSQL datastore with a synchronous standby, and the UpstreamAuthority plugin for Vault-compatible PKI pointed at its cluster's OpenBao mount. spire-controller-manager creates entries from a ClusterSPIFFEID template. The SPIFFE CSI driver mounts the Workload API socket into each pod. Server audit logging is enabled and shipped to ClickHouse as issuance records, with a nightly reconciliation against the datastore.

| Option | Verdict | Reasoning |
|---|---|---|
| SPIRE with node and workload attestation | Chosen | Identity is bound to an attested node and a kubelet-verified process; issuance evidence is first-class; SPIFFE is the standard every mesh reads. |
| istiod's built-in CA with a plugged-in intermediate | Rejected | Simplest, and fast. Puts a signing key inside the configuration compiler and gives weaker attestation evidence. |
| cert-manager with istio-csr and an OpenBao issuer | Rejected | Keeps keys out of istiod and is familiar to Kubernetes teams. Attestation is still token-only, and issuance at 2,000 a second is a load on the Kubernetes API through CertificateRequest objects. |

**What it buys**

- A pod cannot obtain a certificate for an identity other than the one the kubelet reports for it
- istiod holds no signing key; compromising configuration does not mint identities
- The same identities work for non-Kubernetes workloads later, through SPIRE's other attestors

**What it costs**

- A second control-plane component per cluster, with its own datastore to back up
- Istio's SPIRE integration is less common than its built-in CA, so fewer people have debugged it
- Issuance throughput at mass restart depends on the datastore and must be proven (evidence note)

**Choose differently when.** In a single cluster with no audit requirement for issuance evidence and no plan to extend identity beyond Kubernetes, Istio's built-in CA with an intermediate from an offline root gives most of the security for less to operate.

**Why it holds up over time.** SPIFFE identities and the Workload API are standards that Envoy, Istio, Linkerd, Cilium and gRPC all consume. Whatever the mesh becomes, the identity plane can stay, and the thing an auditor asks for (who got which certificate, on what evidence) does not depend on the mesh at all.

> **Lesson.** Keep the component that issues identity separate from the component that distributes configuration, and make issuance evidence something the issuer writes rather than something you reconstruct.

#### ADR-08 · 24-hour workload certificates renewed at 12 hours; short lifetime replaces revocation lists

**Status:** Accepted  ·  **Shown on views:** 09, 16, 21, 06

*How long should a workload certificate live?*

**Context.** Short lifetimes shrink the value of a stolen key and make revocation lists unnecessary; they also multiply issuance load, tighten the clock-skew budget and shorten the control-plane outage the fleet survives. A 1-hour certificate means a 30-minute CA outage is an outage. Long lifetimes invert that and require a working revocation mechanism, which in practice means OCSP or CRL checks on the request path.

**Decision.** Workload SVIDs live 24 hours and are renewed at 12 hours, the requirement's stated assumption, adopted because it produces the 12-hour outage survival the availability target needs. There are no revocation lists. Containment of a compromised identity is: an emergency deny on the principal, banning the node's SPIRE agent so renewal stops, deleting the pod, and waiting at most 24 hours for the certificate to expire. Clock skew is held to ±60 seconds and a node beyond it is tainted.

**How it works on-premise.** SPIRE's default X509-SVID TTL is set to 24 h; agents renew at half-life. chrony on every node syncs to three site-local stratum-1 servers. node-problem-detector reads chrony's offset and taints a node NoSchedule and NoExecute beyond 60 s. Issuance at 40,000 SVIDs a day is about 0.5 a second on average; the design point is the 2,000-a-second restart burst.

| Option | Verdict | Reasoning |
|---|---|---|
| 24 h lifetime, renew at 12 h, no revocation lists | Chosen | 12 hours of issuer outage with no effect, a day's exposure for a stolen key, modest issuance load. |
| 1 h lifetime, renew at 30 min | Rejected | A stolen key is useless in an hour. A 30-minute CA outage becomes a fleet outage, and issuance load is 24 times higher. |
| 30-day lifetime with OCSP or CRLs | Rejected | Negligible issuance load. Revocation checking on the request path is exactly the dependency ADR-01 forbids, and without it a stolen key lives a month. |

**What it buys**

- The fleet survives 12 hours of total issuer outage with no request failures
- No revocation infrastructure to operate, and nothing on the request path to check it
- Clock skew is a non-issue at ±60 s against a 24-hour window

**What it costs**

- A stolen key is valid for up to 24 hours; containment relies on authorisation deny and pod deletion in that window
- A CA outage longer than 24 hours is a fleet outage, so the emergency issuance path must be drilled
- Lifetime and survivable outage cannot be changed independently

**Choose differently when.** Workloads handling highly sensitive keys (a signing service, a payment HSM proxy) should run shorter lifetimes, such as 4 hours renewed at 2, in their own SPIRE entry, and accept a shorter survivable outage for those workloads only.

**Why it holds up over time.** The record names the coupling explicitly: lifetime, renewal point, outage budget and skew budget are one decision. A future change to any one of them has to be made here, where the other three are visible, which is the only reliable defence against a well-meant change that quietly shortens the fleet's survival.

> **Lesson.** Treat credential lifetime as an availability decision as well as a security one, and write both consequences in the same place.

#### ADR-09 · Offline HSM root, name-constrained per-cluster intermediates in OpenBao, roots-only trust bundle

**Status:** Accepted  ·  **Shown on views:** 09, 17, 22, 29

*How does one mesh-wide root become twenty clusters, none of which can mint identities for another?*

**Context.** A single mesh-wide root makes cross-cluster verification simple. If every cluster's issuer can sign any identity, compromising one cluster compromises the mesh. The requirement asks for a mesh-wide root, a per-cluster intermediate with independent lifetime and revocation, the root key offline or in an HSM with a rotation ceremony, reissue of an intermediate within 30 minutes, and a root rotation that is additive with a 30-day overlap.

**Decision.** The root key is generated in and never leaves an HSM kept offline in a vault room, used once a year with two keyholders present to sign intermediates. Each cluster has a one-year intermediate with an X.509 name constraint that permits only its own trust domain, held in a dedicated PKI mount in its site's OpenBao. SPIRE in that cluster obtains a 7-day signing CA from that mount. Proxies trust roots only. Each site holds one pre-signed spare intermediate per cluster, sealed, so an intermediate can be replaced in under 30 minutes without a root ceremony.

**How it works on-premise.** The ceremony runs on an air-gapped workstation using OpenSSL with a PKCS#11 provider against the HSM, from a scripted runbook; outputs are signed intermediates and an updated bundle carried on write-once media. OpenBao runs five Raft voters across three rooms per site, auto-unsealed from a hardware-backed key, with one PKI mount and one SPIRE role per cluster. Root rotation publishes the new root into the SPIRE bundle 30 days before any intermediate chains to it (view 17).

| Option | Verdict | Reasoning |
|---|---|---|
| Offline root, name-constrained per-cluster intermediates, roots-only bundle | Chosen | Cryptographic containment per cluster, simple verification everywhere, and intermediates that rotate without touching bundles. |
| Separate root per cluster with explicit bundle federation | Rejected | Strongest isolation. Twenty roots to rotate and a bundle matrix to distribute for every cross-cluster call. |
| Online root in OpenBao | Rejected | Operationally easy. A network-reachable root is the one key whose compromise cannot be contained. |
| Trust bundle containing intermediates | Rejected | Lets a proxy verify shorter chains. Every intermediate rotation then becomes a fleet-wide bundle change. |

**What it buys**

- A compromised cluster can mint only its own identities, even though all chains end at one root
- OpenBao can be unavailable for three days before any SPIRE CA expires
- Intermediate replacement is a local operation with a sealed spare, not an emergency ceremony

**What it costs**

- Relies on Envoy enforcing URI name constraints, which must be proven on the chosen release
- The yearly ceremony needs two keyholders physically present and a rehearsed runbook
- A sealed spare per cluster per site is key material that must be inventoried and audited

**Choose differently when.** If name-constraint enforcement cannot be proven, keep one root but enforce trust domain in every authorisation policy (principals include the cluster), and plan a move to per-cluster roots with SPIFFE federation for clusters with different risk profiles.

**Why it holds up over time.** Hierarchy depth and lifetimes are arranged so that the rarest operation (root rotation, once a decade) is the only one that touches every proxy, and it is additive. Everything that changes more often happens below the bundle. That shape survives changes of HSM vendor, PKI software and mesh.

> **Lesson.** Arrange a certificate hierarchy so that the operations you do often never touch the thing every client trusts, and the one you do rarely is rehearsed.

### Transport and authorisation

*How a fleet moves from plaintext to strict, where the allow-or-deny decision is made, how a policy is proven before it bites, and what leaves the mesh.*

#### ADR-10 · Move from permissive to strict per service, measured, by reviewed change

**Status:** Accepted  ·  **Shown on views:** 04, 29, 25

*How does a fleet of 2,000 services stop accepting plaintext without an outage?*

**Context.** Turning on strict mTLS everywhere at once breaks every caller that is not yet meshed: batch jobs, legacy VMs, health probes from outside the mesh. Leaving everything permissive forever means the encryption promise is not kept. The requirement asks for permissive mode per service, strict mode per service, namespace or mesh, a per-service plaintext report, and a transition that is a reviewable change rather than a switch.

**Decision.** Every namespace starts permissive when meshed. A service moves to strict through a PeerAuthentication change in the security repository, reviewed by its owners, after the plaintext report has shown zero plaintext for seven days. Namespace-wide strict is allowed only when every service in it is already strict. Mesh-wide strict is not a goal date; it is the state reached when the report shows no permissive services left.

**How it works on-premise.** Istio's standard metrics label every inbound request with its connection security policy (mutual TLS or none). A Thanos recording rule computes plaintext share per destination service daily, and the report lists plaintext sources as workload identities where the source is meshed and as 'no identity' with namespace and pod otherwise. Health probes are rewritten by the injector to go through the sidecar so they do not appear as plaintext.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-service strict after a measured zero, by reviewed change | Chosen | Each flip is backed by data about that service's actual callers and can be reverted in one operation. |
| Mesh-wide strict on a date | Rejected | Fast and simple to communicate. Every unknown caller becomes an incident on the same day. |
| Permissive indefinitely, encryption reported only | Rejected | No breakage. Plaintext stays accepted wherever nobody chooses to act. |

**What it buys**

- Each strict flip is justified by a week of evidence for that service
- Unknown callers are found by the report, not by the outage
- Progress toward full strict mode is measurable and visible per team

**What it costs**

- Full strict mode takes months, and permissive services accept plaintext until then
- The plaintext report depends on telemetry, which is loss-tolerant; zero means zero observed
- Teams must act; a platform team cannot finish the migration alone

**Choose differently when.** A greenfield estate with no unmeshed callers should start strict in every namespace and skip permissive entirely; the migration machinery exists for estates that were not born inside a mesh.

**Why it holds up over time.** The mechanism, observe then enforce per unit, applies unchanged to every future tightening: TLS versions, cipher suites, or post-quantum key exchange when the proxies support it. The report and the reviewed flip become the platform's general tool for raising a floor without an outage.

> **Lesson.** Raise a security floor one tenant at a time on evidence about that tenant, and make the evidence the gate rather than the calendar.

#### ADR-11 · Authorisation is compiled into the destination proxy; external authorisation is a per-service opt-in

**Status:** Accepted  ·  **Shown on views:** 11, 30, 29

*Where is the allow-or-deny decision evaluated?*

**Context.** In-proxy evaluation costs microseconds and cannot consult anything the configuration does not already contain; a policy change is a configuration push with that latency. An external authoriser can use live state such as a session or a risk score, and puts a network call, its latency and its availability into every request, forcing a fail-open or fail-closed choice with no good answer. The requirement sets 0.2 ms p99 for the in-proxy decision and asks for external hooks only as opt-in with a declared budget and posture.

**Decision.** Authorisation policies are Istio AuthorizationPolicy objects compiled into Envoy's RBAC filter at the destination proxy, matching on the authenticated peer identity, namespace, method, path and declared request attributes, never on IP address. In Phase 3, a service may add Envoy's external authorisation filter pointing at an OPA sidecar or a service, declaring in the security repository its timeout, whether it fails open or closed, and which paths it covers.

**How it works on-premise.** Policies are evaluated in Istio's order: custom (external), then deny, then allow. The external option runs OPA with the Envoy plugin as a sidecar, so the network call is to localhost and its failure domain is the pod; policy bundles for OPA come from Harbor. A service's declaration is validated at admission: no external authz filter without a declared timeout under 5 ms and an explicit failure posture.

| Option | Verdict | Reasoning |
|---|---|---|
| In-proxy RBAC, external authz opt-in per service | Chosen | Microsecond decisions with no dependency for the common case; live-state decisions only where a service has chosen to pay for them. |
| External authoriser for every request | Rejected | One policy engine with full context. Puts a network call and its failure mode on every request in the fleet. |
| Authorisation in the application only | Rejected | Full context and no mesh dependency. The requirement is that caller identity be a verifiable input to a decision teams do not have to implement. |
| Network policy only (L3/L4) | Right elsewhere | Right as a second layer (ADR-13). Cannot see method or path, and pod identity by label is weaker than by certificate. |

**What it buys**

- Authorisation adds no network dependency and stays within 0.2 ms p99
- Policies survive rescheduling and IP reuse because they name identities
- Services that need live context can have it without imposing it on the fleet

**What it costs**

- A policy change takes a configuration push to land, typically seconds, never while istiod is down
- Mesh policy cannot express per-user or per-session rules
- Services that opt into external authz own the choice between fail-open and fail-closed

**Choose differently when.** If most services need decisions on per-user entitlements (a multi-tenant SaaS where every call is tenant-scoped), invest in a local external authoriser with pushed data, such as OPA sidecars fed by bundles, as the default, and accept a pod-local dependency on every request.

**Why it holds up over time.** Identity-based, destination-side authorisation is the model the Gateway API policy work and every major mesh share. Policies expressed on SPIFFE identities and request attributes carry across proxies, and the opt-in seam for live-state decisions is Envoy's external authorisation protocol, which is widely implemented.

> **Lesson.** Make the common decision local and free, and make the expensive one explicit, per service, with its failure mode written next to it.

#### ADR-12 · Default deny in strict mode, dry-run before enforcement, and an impact gate at merge

**Status:** Accepted  ·  **Shown on views:** 04, 14, 30

*How is an authorisation policy proven correct before it can deny real traffic?*

**Context.** A policy that denies a legitimate caller is an outage that looks like a security success. Default deny makes that risk larger, because forgetting an allow is now fatal. The requirement asks for default deny in strict mode with explicit allows, a dry-run posture reporting what would have been denied, and refusal of any change that would deny more than a declared share of current traffic without an override.

**Decision.** When a service goes strict, its namespace receives an empty-rule allow policy that denies everything not explicitly allowed; the service's allow policy must merge in the same change. Any new or changed policy is first merged in dry-run mode and runs against live traffic for at least 24 hours. At merge, CI replays the last 24 hours of that service's access logs against the proposed policy; if it would deny more than 0.5% of requests, the merge needs an override approved by a second security owner.

**How it works on-premise.** Dry-run uses Istio's istio.io/dry-run annotation, which compiles the policy into Envoy as shadow rules and records would-be denials in access logs and metrics without denying. The impact replay is a small Go tool in the security repository's pipeline that queries ClickHouse for the service's inbound requests (peer identity, method, path) and evaluates them against the proposed rules with the same semantics. Overrides are recorded as a signed commit trailer.

| Option | Verdict | Reasoning |
|---|---|---|
| Default deny, dry-run, replay gate | Chosen | Two independent proofs, one against history and one against live traffic, before a policy can deny anything. |
| Default allow with deny lists | Rejected | No outages from forgotten allows. Anything nobody thought to deny is permitted, which is the opposite of the requirement. |
| Enforce immediately, roll back on complaints | Rejected | Fastest. Makes callers discover policy errors through failed requests. |

**What it buys**

- A policy that would break a real caller is caught before merge or during dry-run
- The would-deny list names callers by identity, which turns a policy review into a caller review
- The 0.5% threshold makes large-impact changes a deliberate two-person decision

**What it costs**

- A day of dry-run delays every enforcement change except emergency denies
- The replay depends on 24 hours of access logs, which are sampled for successes (ADR-29); rare callers can be missed
- Istio marks dry-run as experimental; its behaviour must be re-verified on each upgrade

**Choose differently when.** For an emergency deny against a compromised identity, skip dry-run and the gate entirely: the change type is labelled emergency, applied to all clusters at once and reviewed after the fact.

**Why it holds up over time.** Shadow evaluation and replay against recorded traffic are techniques, not features of one mesh. If the dry-run annotation disappears, the replay gate still stands, and the platform's promise that no policy bites before it is proven does not change.

> **Lesson.** Test a rule that can deny traffic against the traffic it will see, both what happened yesterday and what happens today, before it is allowed to deny.

#### ADR-13 · Egress only to declared hosts through an egress gateway, with Cilium as the second enforcement plane

**Status:** Accepted  ·  **Shown on views:** 07, 29, 25

*How does traffic leave the mesh, and what stops a workload going around its proxy?*

**Context.** A sidecar can be bypassed by a process that has the privileges to change its pod's network rules, and an undeclared external destination is a common sign of misconfiguration or compromise. The requirement asks for egress to declared destinations through a controlled path, and for reporting, optionally refusing, traffic to undeclared ones.

**Decision.** External destinations are declared as ServiceEntry objects in the routing repository and reached through a per-cluster egress gateway that originates TLS. Strict namespaces run with outbound traffic restricted to the registry; permissive ones allow any destination and report undeclared ones. Cilium network policy independently allows egress from workload namespaces only to in-cluster destinations and the egress gateway, so bypassing the sidecar does not reach the outside.

**How it works on-premise.** Undeclared destinations appear in Envoy metrics under the passthrough and blackhole clusters and in access logs with the original destination; an alert names the source identity. Pod Security Admission at the restricted level denies NET_ADMIN and privileged containers in workload namespaces. The egress gateway runs on the gateway node pool with its own identity, and its outbound source addresses are the only ones perimeter firewalls allow.

| Option | Verdict | Reasoning |
|---|---|---|
| Declared hosts via egress gateway, Cilium as backstop | Chosen | One path out, visible and auditable, with a second plane that does not depend on the sidecar being intact. |
| Sidecar-only egress control | Rejected | Simpler. Any process that can alter the pod's traffic redirection escapes it. |
| Perimeter firewall only | Rejected | Already exists. It sees node addresses, not workload identities, so it cannot say which service called out. |

**What it buys**

- Every external call is attributable to a workload identity
- A bypassed or compromised sidecar still cannot reach undeclared external hosts
- Perimeter firewall rules shrink to the egress gateways' addresses

**What it costs**

- Every new external dependency needs a declaration merged before it works in strict namespaces
- The egress gateway is a shared component whose capacity must follow external call volume
- Two policy systems (Istio and Cilium) must agree about what is allowed out

**Choose differently when.** Where workloads legitimately call thousands of changing external hosts (a web crawler, a webhook delivery service), put them in a dedicated namespace with their own egress path and domain-level filtering rather than individual ServiceEntries.

**Why it holds up over time.** Two independent enforcement planes (identity-aware L7 and label-aware L3/L4) protect against the failure of either, including a future where one of them is replaced. The declarations are plain Kubernetes objects in Git, so the list of what the organisation talks to outlives the tools that enforce it.

> **Lesson.** Control the way out in two places that fail independently, and keep the list of allowed destinations as reviewed data.

### Traffic and resilience

*How routes are written, who owns a weight, how fast a mistake is undone, and how retries and overload are kept from becoming outages.*

#### ADR-14 · Routing intent is written in the Kubernetes Gateway API; Istio APIs only where the standard has no field

**Status:** Accepted  ·  **Shown on views:** 05, 13, 28

*In what language do 250 teams declare their routes?*

**Context.** Routes are the mesh configuration teams write most often and change most. Istio's VirtualService and DestinationRule are expressive and specific to Istio. The Kubernetes Gateway API, through its mesh (GAMMA) profile, is a multi-implementation standard for service-to-service routing that Istio, Linkerd, Cilium and others implement, with weighted backends, header matches and request mirroring. Istio's per-node L7 proxies (waypoints) are configured only through it.

**Decision.** Routes, weights, header cohorts, mirroring and timeouts are written as Gateway API HTTPRoute (and GRPCRoute) objects attached to a Service. Subsets are separate Kubernetes Services selecting a version label, referenced as weighted backends. Connection pool, outlier detection, locality and TLS settings, which the standard does not yet cover, are written as Istio DestinationRules. Every Istio-specific object is listed in an allowlist in the routing repository with the Gateway API issue that would replace it.

**How it works on-premise.** Gateway API CRDs are installed per cluster at a pinned standard-channel version matching the Istio release. A generator in the routing repository creates per-version Services from a short service manifest, so teams write one file per service. Admission guards reject VirtualService objects outside the allowlist, so the two APIs cannot both route the same service.

| Option | Verdict | Reasoning |
|---|---|---|
| Gateway API first, Istio APIs for gaps | Chosen | Portable intent for the objects teams touch most, full capability for the rest, and the path Istio's own per-node mode requires. |
| Istio VirtualService and DestinationRule throughout | Rejected | Most expressive today and most documented. Ties every team's routes to one mesh's API. |
| A platform-defined abstraction CRD that generates both | Rejected | Smallest interface for teams. Creates an in-house API with no community, documentation or tooling, owned by us forever. |

**What it buys**

- The routes teams write are portable across mesh implementations
- Moving a namespace to per-node L7 proxies later needs no route rewrite
- Gateway API's role separation (route owners versus infrastructure owners) matches ADR-22

**What it costs**

- Two APIs to learn during the transition, and an allowlist to maintain
- Some fields (retry details, mirroring percentages) sit in the experimental channel on current releases
- Per-version Services add objects per service, generated rather than hand-written

**Choose differently when.** If the estate will never run another mesh and needs features only VirtualService offers today (complex fault injection, rich retry predicates) across most services, VirtualService throughout is simpler for the next three years.

**Why it holds up over time.** Standards absorb the features that most implementations agree on, so the allowlist of Istio-specific objects should shrink with each Gateway API release. Every entry names what would retire it, which turns API drift into a tracked backlog rather than a surprise migration.

> **Lesson.** Write the configuration you change most in the most widely implemented language available, and keep an explicit, shrinking list of where you could not.

#### ADR-15 · The rollout controller owns traffic weights during a rollout, and judges the share the mesh reports

**Status:** Accepted  ·  **Shown on views:** 05, 18, 28

*Who owns the traffic weights while a release is rolling out?*

**Context.** The candidates are the mesh API as system of record with the pipeline writing to it, a rollout controller that owns and reconciles weights, or the CD system writing weights directly. Each puts the authority to roll back somewhere different. During a partial failure, if two systems both believe they own a weight, the rollback races the rollout. The requirement also asks that the delivery system read effective weights from the mesh rather than assume its intent took effect.

**Decision.** Argo Rollouts owns the weights of a route for the duration of a rollout. It writes weight steps to the HTTPRoute, runs analysis against Thanos on the canary's reported share, error ratio and latency relative to stable in the same window, and aborts automatically on breach. A human stops a release by aborting the Rollout, never by editing the route. Outside a rollout, the routing repository is the owner.

**How it works on-premise.** The Rollout object references the HTTPRoute through the Gateway API traffic-router plugin. Admission guards deny updates to a route carrying an active-rollout annotation from any identity other than the Argo Rollouts controller's service account. Analysis templates are shared from the platform: applied share within ±20% of target, 5xx ratio no worse than 1.5× stable, p99 no worse than 1.3× stable, with a minimum request count per step.

| Option | Verdict | Reasoning |
|---|---|---|
| Rollout controller owns weights; analysis reads mesh telemetry | Chosen | One writer during a rollout, an abort that is fast and in one place, and a judgement based on what the mesh did. |
| CD pipeline writes weights directly | Rejected | No extra controller. A pipeline job that dies mid-rollout leaves weights with no owner, and its abort path is a second job. |
| Git as system of record for every weight step | Rejected | Perfect audit trail. A rollback becomes a commit plus a sync, which is minutes when the requirement asks for under 60 seconds. |

**What it buys**

- Rollback and rollout cannot race, because only one identity may write the route
- An abort is one command against one object and takes effect in seconds
- A weight change that never landed is detected before a canary is judged

**What it costs**

- During a rollout the live route differs from Git; the route's intent sha reflects the Rollout object, not a commit
- Argo Rollouts is on the path of every release, so its availability matters on release days
- Low-traffic services need longer steps to reach the minimum request count

**Choose differently when.** Where releases are rare and manual and there is no SLO-driven analysis, weights written through Git with a normal review are simpler, and the 60-second rollback target should be renegotiated rather than faked.

**Why it holds up over time.** The principle, one writer per mutable field and judgement on observed state, is independent of Argo Rollouts. Any replacement controller inherits the same admission rule and the same analysis contract, and the handover between Git ownership and rollout ownership is an annotation any tool can respect.

> **Lesson.** Give every mutable field exactly one writer at any moment, and judge an action by what the system reports happened, not by what you asked for.

#### ADR-16 · Revert pins a previously rendered bundle, and is the one operation allowed ahead of Git

**Status:** Accepted  ·  **Shown on views:** 14, 23, 12

*How is reverting a mesh change made the fastest path in the system?*

**Context.** In a GitOps flow, undoing a change means a revert commit, a review, a render and a sync, which is minutes. The requirement asks for revert to a previously applied configuration as a single operation and for revert to be the fastest path, with automated rollback inside 60 seconds of a breach. A revert that waits on the same pipeline as the change it undoes is as slow as the change.

**Decision.** Every merged change is rendered once into a signed OCI bundle in Harbor, and each cluster records which bundle digest it has applied. Revert is a single command (or an automatic action from a wave gate) that pins the affected clusters to the previous digest: it suspends Argo CD's automatic sync for that application, applies the stored bundle directly, and opens the Git revert commit afterwards. Once the revert commit merges, sync resumes and the pin is released.

**How it works on-premise.** Bundles are Kubernetes manifests packaged with ORAS, signed with cosign using a key in OpenBao, and verified by admission before apply. The revert command is a small controller with its own identity whose only permissions are to suspend sync and apply a verified bundle to mesh resource kinds. Every revert writes an audit event with the from and to digests; a pin older than two hours pages the owning team.

| Option | Verdict | Reasoning |
|---|---|---|
| Pin a pre-rendered signed bundle, then catch Git up | Chosen | Seconds, not minutes; nothing is recompiled; Git remains the record once the revert commit lands. |
| Git revert and normal sync | Rejected | Pure GitOps. Slower than the change it undoes, and depends on the review and sync path being healthy during an incident. |
| kubectl apply of the previous version by hand | Rejected | Fast. Unverified, unaudited, and Argo CD reapplies the bad version at its next sync. |

**What it buys**

- Rollback meets 60 seconds without depending on CI, review or rendering
- The reverted state is exactly what was running before, verified by signature
- Argo CD cannot fight the revert, because its sync is suspended for the duration

**What it costs**

- For a short window the cluster is ahead of Git, which is a deliberate and alerted exception to GitOps
- Bundle storage and signing add a component to the change path
- A revert of a change that also altered non-mesh resources reverts only the mesh kinds

**Choose differently when.** For platforms whose changes are rare and whose rollback target is measured in tens of minutes, a plain Git revert is simpler and keeps the invariant that the cluster never leads Git.

**Why it holds up over time.** Content-addressed, signed artefacts are how software supply chains are moving in general, so the bundle store is not a mesh-specific invention. The rule that the undo path must not share dependencies with the do path applies to whatever replaces Argo CD.

> **Lesson.** Make undo independent of the machinery that produced the mistake, and keep what you would undo to already built and signed.

#### ADR-17 · Timeouts are mandatory, retries are budgeted, and deadlines travel with the call

**Status:** Accepted  ·  **Shown on views:** 11, 20, 31

*How do proxy retries hide a transient failure without turning a slow dependency into a fleet-wide outage?*

**Context.** Retries in a proxy are a shared amplifier. Three attempts per call triple the load on a dependency that is already failing, and nested retries multiply: three hops each retrying three times send 27 requests to the bottom of the stack. The requirement asks for a per-route timeout with no route allowed without one, retries only for declared idempotent methods and conditions with bounded attempts, per-try timeout and jitter, a retry budget per destination, and deadlines that bound nested retries.

**Decision.** Every route declares a timeout; admission rejects a route without one, and the platform default applied by the generator is 2 s. Retries apply only to GET, HEAD and methods a service declares idempotent, on connection failure, reset and 503, with at most three attempts, a per-try timeout and 25 ms base backoff with full jitter. Each destination has a retry budget of 20% of active requests. The caller's remaining deadline travels as a header; the inbound proxy caps its route timeout by it, and no retry starts if the remaining deadline cannot cover a per-try timeout.

**How it works on-premise.** Timeouts and retry policy are set on HTTPRoute (or DestinationRule where the standard lacks the field). Envoy's retry_budget in cluster circuit breakers and the router's respect_expected_rq_timeout option are not exposed by Istio's API on every release, so they are applied by one EnvoyFilter generated per revision, owned by the platform team and exercised by the upgrade conformance suite. Applications forward the deadline header alongside trace context through the same OpenTelemetry propagator configuration.

| Option | Verdict | Reasoning |
|---|---|---|
| Mandatory timeouts, budgeted retries, propagated deadlines | Chosen | Bounds amplification at every level: per call, per destination and per call chain. |
| Attempt counts only | Rejected | Available through the standard API today. Bounds one call and does nothing about a thousand callers retrying together. |
| No proxy retries; applications decide | Rejected | No amplification from the mesh. Gives up the requirement's transparent handling of transient failure. |

**What it buys**

- A widespread failure increases load on the failing service by at most a fifth
- Nested calls cannot multiply retries past the original caller's deadline
- No route in the mesh can wait forever

**What it costs**

- Two features depend on an EnvoyFilter, which is a documented upgrade risk
- Deadline propagation needs every application to forward one header
- Some legitimate slow endpoints need explicit longer timeouts, which teams must declare

**Choose differently when.** Where call chains are shallow (one or two hops) and traffic is low, attempt counts with jittered backoff are enough, and the EnvoyFilter is not worth its upgrade risk.

**Why it holds up over time.** Budgets and deadlines are properties of the traffic, not of Istio; gRPC, Envoy and every mature RPC stack implement them. When the standard API exposes the budget, the EnvoyFilter is deleted and nothing about the behaviour changes.

> **Lesson.** Bound retries by a fraction of real traffic and by the caller's deadline, not only by an attempt count, or the mesh becomes the amplifier in every outage.

#### ADR-18 · Outlier ejection and load shedding at the proxy; locality routing with a declared spillover threshold

**Status:** Accepted  ·  **Shown on views:** 11, 19, 20

*How does the mesh handle a bad instance, an overloaded destination and a failing zone?*

**Context.** A single bad pod should leave the load-balancing set quickly and come back cautiously. An overloaded destination should reject fast instead of queueing until every caller times out. Nearest-zone routing saves latency and cross-zone cost, but the point at which traffic spills to another zone or cluster is a cost and blast-radius decision that defaults hide. The requirement asks for ejection with bounded, escalating return, per-destination concurrency and pending limits, and a declared spillover threshold.

**Decision.** Every destination has outlier detection: eject after five consecutive 5xx or gateway errors, base ejection 30 s multiplied by the number of prior ejections up to 5 minutes, and at most 50% of endpoints ejected. Connection pools cap connections, pending requests and active requests per destination, with pending requests limited so the callee's proxy sheds with a fast 503 marked as overflow. Locality-aware routing prefers the caller's room, then its site, then remote clusters, and spills over when fewer than 70% of the preferred locality's endpoints are healthy, declared per destination.

**How it works on-premise.** Settings live in DestinationRules generated from the service manifest with platform defaults and per-service overrides. Locality comes from standard topology labels on nodes (room as zone, site as region). Envoy's overprovisioning factor is set per destination so that the 70% threshold is the number written in the manifest, not an implicit 1.4 factor. Spillover requests are counted by source and destination locality for the cost report.

| Option | Verdict | Reasoning |
|---|---|---|
| Proxy-side ejection, shedding and declared locality spillover | Chosen | Uniform behaviour for every service, visible thresholds, and overload refused where it is cheapest. |
| Client libraries with circuit breakers | Rejected | Rich per-call context. Different behaviour per language and per library version across 250 teams. |
| Round-robin across all localities | Rejected | Even load. Pays cross-room latency and cost on every request and spreads a failing zone's errors everywhere. |

**What it buys**

- Bad pods leave rotation in seconds and repeat offenders stay out longer
- Overload produces fast, attributed failures instead of timeouts
- Cross-zone and cross-cluster traffic happens only when a declared threshold is crossed

**What it costs**

- Aggressive ejection on a small pool can remove healthy capacity; the 50% cap is the guard
- Limits set too low shed traffic a service could have handled; they need load testing per service
- Locality routing concentrates load in one room when callers are unevenly placed

**Choose differently when.** For small services with two or three pods, disable percentage-based ejection and rely on health checks, because ejecting one pod removes a third of capacity.

**Why it holds up over time.** Ejection, concurrency limits and locality are standard load-balancer behaviour in every proxy generation. Declaring thresholds per destination, in reviewed data, means they survive a change of proxy and remain visible to whoever inherits the service.

> **Lesson.** Make every threshold that decides where traffic goes a declared number, because an implicit default is a decision nobody remembers making.

### Multi-cluster and network

*How traffic crosses a cluster or a site, how trust between clusters is declared, and what the network under the mesh provides.*

#### ADR-19 · Flat pod networking inside a site; east-west gateways between sites

**Status:** Accepted  ·  **Shown on views:** 19, 22, 10

*How does traffic cross a cluster boundary?*

**Context.** A flat network, with every pod IP routable and remote endpoints in local discovery, gives the shortest path and the simplest routing, and requires non-overlapping address space and a posture where one cluster's compromise reaches every pod at L3. East-west gateways expose each cluster through one hop, add latency and a component, contain blast radius, and work across networks never designed to be joined. Here the network inside each site is under the platform team's control; links between sites are not.

**Decision.** Clusters within one site share a routable pod network with non-overlapping CIDRs, and calls between them go pod to pod with mTLS. Calls between sites go through east-west gateways that route on SNI in passthrough mode, so TLS from the caller's proxy reaches the destination proxy intact and the gateway holds no workload identity. The MVP is single-site, which matches the requirement's single-region MVP.

**How it works on-premise.** Each site allocates a /14 of pod addresses and assigns each cluster a non-overlapping block, advertised by Cilium over BGP to the site's leaf switches. Cilium network policy denies cross-cluster traffic by default except between meshed namespaces. East-west gateways run as Envoy in Istio's AUTO_PASSTHROUGH mode on a dedicated gateway node pool, behind a BGP-announced virtual address with ECMP across gateway pods.

| Option | Verdict | Reasoning |
|---|---|---|
| Flat inside a site, gateways between sites | Chosen | Shortest path where the network is ours and failure domains are shared anyway; containment where links cross sites. |
| Flat everywhere | Rejected | Simplest routing. Needs routed pod addresses across sites and lets a compromise in one site reach every pod in another. |
| Gateways everywhere, even inside a site | Rejected | Uniform model. Adds a hop and a component to traffic that already shares switches, power and failure domain. |

**What it buys**

- Most cross-cluster traffic, which stays inside a site, pays no extra hop
- Cross-site traffic is contained to named gateway addresses that firewalls and capacity plans can see
- Caller identity reaches the destination intact across sites

**What it costs**

- Two cross-cluster models to operate and debug
- Pod address planning per site is a prerequisite and hard to change later
- Gateways are capacity to size for cross-site failover, not for normal traffic

**Choose differently when.** If the network inside a site is owned by another organisation, or clusters inside a site belong to different security zones (a PCI enclave, for example), put gateways between those clusters too.

**Why it holds up over time.** The split follows ownership of the network, which changes slowly. Adding a site adds gateways without touching existing sites, and SNI passthrough preserves end-to-end identity regardless of which proxy implementation runs in the gateway.

> **Lesson.** Join networks directly only where you own both ends and share a failure domain; everywhere else, make the crossing a named, narrow, identity-transparent hop.

#### ADR-20 · Cross-cluster trust and service export are declared, never default

**Status:** Accepted  ·  **Shown on views:** 19, 09, 12

*When does one cluster accept another cluster's identities and see its services?*

**Context.** Multi-cluster meshes commonly merge every service from every cluster into every other cluster's discovery, and accept every identity chained to the shared root. That is convenient and means a cluster's full service list and every workload identity are trusted everywhere without anyone having decided so. The requirement asks for trust accepted only by declared federation, discovery partitioned per cluster, and replication of only what federation declares.

**Decision.** Each service that should be reachable from other clusters is exported explicitly in the routing repository, naming the clusters allowed to import it. Each cluster's authorisation policies accept identities from other trust domains only where the security repository lists them for that destination. A cluster with no declarations sees and trusts only itself, even though it shares the mesh root.

**How it works on-premise.** Exports are a label on the Service plus an entry in a federation manifest that the routing generator turns into istiod discovery selectors and remote-secret scope for peer clusters. Trust is expressed as principals in AuthorizationPolicy including the peer's trust domain (for example c12.mesh.corp/ns/shop/sa/checkout), so an unlisted cluster's identity fails authorisation even with a valid chain.

| Option | Verdict | Reasoning |
|---|---|---|
| Explicit export and explicit trust per destination | Chosen | Discovery and trust grow only by reviewed decisions, and each is visible in Git. |
| Merge all services, trust the whole root | Rejected | Zero configuration. Every cluster's compromise and every cluster's naming mistake is shared. |
| Separate meshes joined by gateways with translation | Right elsewhere | Right between organisations or companies. Loses end-to-end identity, which this design needs inside one organisation. |

**What it buys**

- Losing or compromising a cluster exposes only what it exported and whom it was trusted by
- Per-cluster discovery stays small, which keeps istiod and proxy memory bounded
- A reviewer can answer 'who can call this from another cluster' from Git

**What it costs**

- Every cross-cluster dependency needs two declarations, export and trust
- Failover to a cluster that did not export a service does not happen, which must be caught in review
- Discovery selectors and remote secrets need tooling to stay consistent with the manifest

**Choose differently when.** In a small mesh of identical, interchangeable clusters (one application deployed to three clusters for capacity), a namespace-level export of everything is simpler and the review burden buys little.

**Why it holds up over time.** Declared federation is the SPIFFE model and the direction multi-cluster service APIs in Kubernetes take. The manifest is plain data, so it can drive a future mesh's multi-cluster mechanism without re-deciding what is shared.

> **Lesson.** Sharing a root is a cryptographic convenience; trust should still be something each destination grants on purpose.

#### ADR-21 · Cilium for pod networking, BGP-announced gateway addresses and a second enforcement plane

**Status:** Accepted  ·  **Shown on views:** 07, 08, 22

*What provides the network under the mesh on hardware without a cloud load balancer?*

**Context.** The requirement's cloud stack puts network load balancers in front of east-west gateways. On-premise, the equivalent must come from the cluster itself or from physical load balancers. The mesh also needs a network layer that can block traffic independently of the mesh during an incident (ADR-04, ADR-13), and routable pod addresses within a site (ADR-19).

**Decision.** Cilium is the CNI on every cluster, in eBPF mode with kube-proxy replacement. Its BGP control plane peers with the site's leaf switches to advertise pod CIDRs and the virtual addresses of east-west and egress gateways, allocated by its load-balancer IPAM, with ECMP across gateway pods. CiliumNetworkPolicy provides default-deny between namespaces at L3/L4 as the second enforcement plane.

**How it works on-premise.** Each node peers with two leaf switches. Gateway virtual addresses are announced only from nodes running a ready gateway pod, so a failed node withdraws its route in seconds. Cilium's own L7 and mutual-authentication features are left off to keep a single L7 enforcement point (Istio) and avoid two proxies on the same connection.

| Option | Verdict | Reasoning |
|---|---|---|
| Cilium with BGP and LB IPAM | Chosen | CNI, virtual addresses and L3/L4 policy from one component with its own agent per node. |
| Calico with MetalLB | Rejected | Mature and widely run. Two components for what Cilium does in one, with no eBPF service load balancing. |
| Hardware load balancers in front of gateways | Rejected | Familiar to network teams. A ticket-driven change for every new gateway address and a device outside the cluster's failure domain. |

**What it buys**

- Gateway addresses follow healthy pods automatically, without a separate device
- An emergency block is available that does not depend on istiod
- kube-proxy's iptables scaling limits are removed at 2,000 pods per cluster

**What it costs**

- Network and platform teams share responsibility for BGP sessions
- Two policy systems (Cilium and Istio) must be kept consistent
- Cilium's kernel requirements constrain the node operating system

**Choose differently when.** If the network team requires all load balancing on dedicated appliances with their own change process, keep appliances for gateway addresses and use Cilium only as CNI and policy engine.

**Why it holds up over time.** BGP is the most stable protocol in the building. Cilium's role here is deliberately narrow (routes, addresses, L3/L4 policy), so replacing it later with another eBPF or BGP-capable CNI changes no mesh configuration.

> **Lesson.** Use the network layer for what it is best at, reachability and coarse policy, and keep it independent of the mesh so each can protect against the other's failure.

### Change and governance

*Where intent lives, who may change which part, how a change and an upgrade reach 40,000 proxies, and what is recorded.*

#### ADR-22 · Intent in three repositories, with separation of duties enforced by repository and by Argo CD project

**Status:** Accepted  ·  **Shown on views:** 13, 29, 04

*How is it enforced, not just agreed, that people who change routing cannot change authorisation or identity?*

**Context.** The requirement asks for separation of duties between routing and authorisation or identity changes, for intent to be versioned immutably and traceable to what each proxy runs, and for strong consistency with full history. Review conventions such as code owners are bypassable by repository administrators and, in some Git services, are paid features. Kubernetes RBAC alone does not know which team a change came from.

**Decision.** Mesh intent lives in three repositories on Forgejo: routing (service teams and the platform team), security (authorisation, peer authentication, trust declarations; security team), and identity (SPIRE templates, intermediates, bundles; PKI owners). Commits are signed. Each repository is applied by its own Argo CD project whose allowed resource kinds match that repository, so the routing project cannot create an AuthorizationPolicy even if one is committed there.

**How it works on-premise.** Forgejo runs on PostgreSQL with a synchronous standby in a second room, with read-only push mirrors at the other two sites. Branch protection requires signed commits and two approvals from the repository's owner team. Argo CD AppProjects list namespaceResourceWhitelist per repository; admission guards reject mesh objects whose manager is not the expected Argo CD project. Every applied object carries the commit sha as an annotation.

| Option | Verdict | Reasoning |
|---|---|---|
| Three repositories, three Argo CD projects with kind allowlists | Chosen | Two independent enforcement layers, neither of which depends on reviewers noticing a file path. |
| One repository with code-owner rules | Rejected | Simpler cross-cutting changes. Enforcement depends on the Git service's paid features and on administrators not bypassing them. |
| Direct kubectl with RBAC per team | Rejected | No GitOps machinery. No review, no signed history, and no link from a proxy's configuration to an intent version. |

**What it buys**

- A routing change cannot alter who may call a service, by construction
- Every applied object is traceable to a signed commit and its approvers
- The auditor's question 'who changed this policy' is a Git log query

**What it costs**

- Changes that need a route and a policy together need two merges in order
- Three repositories and three projects to maintain
- A security-team bottleneck on authorisation changes unless reviews are staffed

**Choose differently when.** A small organisation where the same five people run routing and security gains nothing from the split, and one repository with protected paths is enough.

**Why it holds up over time.** The split follows organisational roles, which change more slowly than tools. Moving from Forgejo to another Git service or from Argo CD to another reconciler preserves the three-repository boundary and its kind allowlists.

> **Lesson.** Enforce separation of duties where changes are applied, not only where they are reviewed.

#### ADR-23 · Argo CD per site, and every mesh change rolls out in cluster waves behind health gates

**Status:** Accepted  ·  **Shown on views:** 23, 14, 22

*How does a valid-looking change that breaks routing avoid breaking it everywhere at once?*

**Context.** A configuration that passes every schema and policy check can still break routing mesh-wide, for example a default timeout that is too short for one heavy path. The requirement asks for validation, staged waves with per-wave health gates, and automatic halt and revert on error-rate breach, and for no cross-region dependency for in-region configuration.

**Decision.** Each site runs its own Argo CD managing only that site's clusters, reading from the local Forgejo mirror. Mesh applications sync in waves: wave 0 is a canary cluster carrying synthetic and a slice of real traffic, wave 1 is one cluster per site, wave 2 is the remaining clusters site by site. A gate between waves waits a fixed soak and checks mesh-attributed error ratio, NACK count and proxies still on the previous sha. On breach it halts and triggers the revert in ADR-16.

**How it works on-premise.** Waves are ApplicationSets with cluster generators ordered by a wave label; the gate is a small controller with read access to Thanos and permission only to advance a wave label or call the revert controller. Argo CD instances run in the site's management cluster with three replicas. Emergency denies are a labelled change type that syncs all clusters without waves.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-site Argo CD, cluster waves, automated gates | Chosen | A bad change reaches one cluster first and is halted by signals the mesh already emits. |
| Central Argo CD for all sites | Rejected | One place to look. A site partition stops changes in the other sites and puts cross-site latency in every sync. |
| Sync everywhere at once, rely on validation | Rejected | Fastest convergence. Validation cannot catch a change that is valid and wrong. |

**What it buys**

- A valid but wrong change affects one cluster before the gate halts it
- Sites change independently during a partition
- Gates use mesh signals, so unrelated application incidents do not block changes

**What it costs**

- A full rollout takes about 45 minutes instead of seconds
- The canary cluster must carry enough real traffic for its signals to mean something
- Emergency changes bypass waves and depend on their own review after the fact

**Choose differently when.** With two or three clusters, waves add little over a canary namespace inside one cluster; stage by namespace rather than by cluster.

**Why it holds up over time.** Progressive rollout of configuration is how large operators have converged on handling configuration outages, and it is independent of the reconciler. The wave labels and gate contract are plain data any delivery tool can respect.

> **Lesson.** Validation proves a change is well formed; only staged exposure proves it is right. Stage configuration the way you stage code.

#### ADR-24 · Proxy configuration is scoped to declared dependencies and distributed as deltas

**Status:** Accepted  ·  **Shown on views:** 14, 26, 27

*What is the unit of configuration distribution?*

**Context.** Full snapshots per proxy on every change are simple and idempotent, and at 40,000 proxies make one route change a fleet-wide push. Deltas cut that by orders of magnitude and introduce sequencing, resynchronisation and silent divergence. Scoping each proxy to the destinations its workload calls helps both and needs a dependency graph, which, if inferred, is wrong exactly when a service starts calling something new. Unscoped, every proxy's memory grows with the whole mesh.

**Decision.** Scope every proxy with an Istio Sidecar resource. In the MVP the scope is the workload's own namespace plus namespaces it declares. In Phase 2 the scope is generated from per-service dependency declarations in the routing repository. Configuration is distributed as delta xDS. A call to an undeclared destination fails with a distinct response flag, is attributed to mesh configuration, and alerts the owning team with the missing declaration.

**How it works on-premise.** The routing generator turns each service manifest's dependsOn list into a Sidecar egress host list. A CI check compares declarations against the last 7 days of access logs and warns on declared-but-unused and on observed-but-undeclared destinations. The drift exporter forces a proxy to resubscribe if its acknowledged sha lags intent by more than 60 seconds, which bounds the delta divergence class of bug.

| Option | Verdict | Reasoning |
|---|---|---|
| Declared scope, delta distribution, drift-bounded | Chosen | Memory tracks what a workload calls, pushes cost what they touch, and divergence is detected within a minute. |
| Full mesh configuration to every proxy | Rejected | Nothing to declare and nothing to go wrong. Memory and push cost grow with the fleet and break the budget well before 40,000 proxies. |
| Scope inferred from observed traffic | Rejected | No declarations to maintain. Refuses a new dependency on the first call, which is the moment nobody is watching. |
| Full snapshots, scoped | Deferred | Idempotent and easier to reason about. Kept as the fallback if delta distribution misbehaves on the chosen release. |

**What it buys**

- Per-proxy memory stays roughly constant as the mesh grows
- A one-service change reaches only the proxies that call that service
- Dependency declarations double as documentation and as input to authorisation reviews

**What it costs**

- Teams must declare dependencies, and a missing declaration refuses calls in scoped namespaces
- Delta distribution adds a class of bug where a proxy silently lags intent
- The generator and the log comparison are platform code to maintain

**Choose differently when.** Below a few hundred services, full configuration to every proxy fits the memory budget, and the declaration burden is not worth it.

**Why it holds up over time.** Scope by declared dependency is independent of the mesh; any xDS control plane, and any per-node data plane, benefits from the same graph. The graph is also the input to future work that has nothing to do with proxies: blast-radius analysis, ownership and change impact.

> **Lesson.** Make what a service depends on a declared, reviewed fact, and let configuration size, policy review and impact analysis all read from it.

#### ADR-25 · Admission guards are the second line: OPA Gatekeeper and ValidatingAdmissionPolicy

**Status:** Accepted  ·  **Shown on views:** 02, 07, 14

*What stops an invalid or unsafe mesh object that never went through CI?*

**Context.** CI checks run on changes that go through Git. Anything applied by kubectl, by a controller, or by a restored backup skips them. Istio accepts many configurations that are valid and unsafe: a route with no timeout, retries on POST, a VirtualService racing a Gateway API route, a route edited while a rollout owns it. The requirement's rules must hold regardless of how an object arrived.

**Decision.** Every rule that CI enforces on mesh objects is also enforced at admission in every cluster. Structural rules that need no external data (timeouts present, retry methods idempotent, sidecar present in strict namespaces) are ValidatingAdmissionPolicies evaluated inside the API server. Rules that need cluster context (an active rollout owns this route, this object's manager matches its repository) are OPA Gatekeeper constraints.

**How it works on-premise.** Constraint templates and admission policies live in the identity-and-platform repository and are applied in wave 0 first. Gatekeeper runs three replicas with failurePolicy Fail for mesh kinds in strict namespaces and Ignore elsewhere, and its audit mode reports existing objects that violate a new constraint before the constraint enforces. The same Rego is used by conftest in CI, so the two lines cannot drift.

| Option | Verdict | Reasoning |
|---|---|---|
| CEL admission policies plus Gatekeeper, same rules as CI | Chosen | No path around the rules, and the simplest rules have no webhook to fail. |
| CI checks only | Rejected | One place to maintain. Anything applied outside Git is unchecked, including during incidents when rules matter most. |
| Kyverno | Rejected | YAML policies many teams find easier than Rego. The organisation's security team already writes Rego for conftest, and one language for both lines avoids drift. |

**What it buys**

- The mesh's safety rules hold for objects from any source
- Most rules run in the API server with no additional availability dependency
- Audit mode shows the impact of a new rule before it blocks anything

**What it costs**

- Gatekeeper is another webhook whose availability matters for strict namespaces
- Rules exist in two forms (CEL and Rego) that must stay consistent
- Overly strict rules block emergency changes unless an audited bypass exists

**Choose differently when.** If every change to every cluster provably goes through one pipeline, with cluster-admin credentials held only by that pipeline, admission guards duplicate CI and can be reduced to the rules about pods.

**Why it holds up over time.** Admission control is part of Kubernetes, and CEL admission policies are the direction the project has taken to reduce webhook dependencies. Rules written against resource fields survive changes of the tools that create those resources.

> **Lesson.** Enforce invariants at the last point every change must pass through, not only at the point most changes pass through.

#### ADR-26 · Proxy fleet upgrades by control-plane revision, never in place

**Status:** Accepted  ·  **Shown on views:** 24, 31, 23

*How are 40,000 proxies upgraded with per-wave verification and an abort that leaves a working fleet?*

**Context.** Upgrading istiod in place means every proxy in the cluster immediately talks to a newer control plane, and rolling back means downgrading a control plane that proxies of a newer version may already be using, which upstream does not support. The requirement asks for upgrades in declared waves with health verification, an abort that leaves a mixed but working version set, and a skew window of at least one minor version in each direction.

**Decision.** A new Istio minor version is installed as a new revision beside the current one. Namespaces move by changing their revision tag, from low-risk to tier-1 in three waves, and their pods restart onto the new proxy. Each proxy connects only to the istiod of its own revision, so a proxy never talks to an older control plane and skew in both directions is held by construction. Abort is retagging back and restarting. The old revision is removed only when no proxy uses it.

**How it works on-premise.** Revision tags (canary, stable) are managed with istioctl tag in the platform repository, applied through Argo CD. Rolling restarts are rate-limited to the SPIRE issuance headroom and gated by PodDisruptionBudgets. The conformance suite, including the retry-budget EnvoyFilter and dry-run policies, runs against each new revision in wave 0 before any namespace moves. The platform upgrades at least quarterly to stay inside upstream support.

| Option | Verdict | Reasoning |
|---|---|---|
| Revision-based canary upgrades | Chosen | Both versions serve in parallel, abort is a retag, and no unsupported version pairing ever exists. |
| In-place control-plane upgrade, then proxy restarts | Rejected | One istiod at a time. Every proxy meets the new control plane at once, and rollback pairs newer proxies with an older control plane. |
| Blue-green whole clusters | Rejected | Cleanest isolation. Needs double capacity per cluster for every upgrade, four times a year. |

**What it buys**

- A problem in a new version affects only the namespaces moved so far
- The fleet can sit mixed for weeks if a wave needs investigation
- Upgrades are frequent and small, which keeps each one cheap

**What it costs**

- Two control planes run per cluster during an upgrade, doubling istiod resources temporarily
- Every upgrade restarts every meshed pod in the cluster
- Teams see pod restarts they did not initiate, four times a year

**Choose differently when.** For patch releases within a minor version, in-place upgrades with a rolling restart are simpler and the skew risk is negligible; reserve revisions for minor versions.

**Why it holds up over time.** Side-by-side versions with traffic moved in waves is the general pattern for upgrading anything on a request path. The quarterly cadence prevents the multi-version jump that turns an upgrade into a project, which is how meshes are most often abandoned.

> **Lesson.** Upgrade by running old and new side by side and moving tenants, and upgrade often enough that each move is small.

#### ADR-27 · An append-only audit trail on Object Lock storage, independent of the SIEM

**Status:** Accepted  ·  **Shown on views:** 12, 13, 03

*Where is the record of who changed what, and who got which certificate, kept so that it cannot be edited?*

**Context.** The requirement asks that every configuration change, policy change and certificate issuance be recorded in an append-only trail with actor, before and after state and time, retained for 13 months. Git history can be rewritten by an administrator, Kubernetes audit logs rotate on the API server, and a SIEM's retention is set by another team's budget.

**Decision.** Three sources feed one trail: Git commits with signatures and approvers, Kubernetes audit events for mesh resource kinds (including before and after objects), and SPIRE issuance records. A collector writes them hourly as compressed files to a Ceph RGW bucket with S3 Object Lock in compliance mode and a 13-month retention. The SIEM receives a copy for correlation but is not the record.

**How it works on-premise.** API server audit policy logs RequestResponse for mesh CRDs and metadata for everything else. The OpenTelemetry Collector routes audit and issuance streams to both ClickHouse (for query) and the Object Lock bucket (for evidence). A daily job verifies the bucket against ClickHouse counts and the Git log and alerts on gaps. Forgejo's own backups also go to an Object Lock bucket.

| Option | Verdict | Reasoning |
|---|---|---|
| Object Lock bucket as record, SIEM as copy | Chosen | Immutable by storage policy, retention set by this platform's requirement, queryable through ClickHouse. |
| SIEM as the record | Rejected | Already exists and is searched by the security team. Retention and immutability are outside this platform's control. |
| Git history alone | Rejected | Captures intent. Misses changes applied outside Git and says nothing about certificate issuance. |

**What it buys**

- The audit trail cannot be altered, even by the platform's administrators, within retention
- Changes applied outside Git appear in the same trail as those that went through it
- Retention is guaranteed by storage policy, not by a scheduled job

**What it costs**

- Compliance-mode Object Lock cannot be shortened, so a wrong retention setting costs storage for 13 months
- API server audit at RequestResponse level for mesh kinds adds load and volume
- A reconciliation job is needed to detect gaps from the loss-tolerant telemetry path

**Choose differently when.** If the organisation's SIEM already provides immutable retention beyond 13 months with guarantees the audit function accepts, write directly to it and drop the separate bucket.

**Why it holds up over time.** S3 Object Lock is an interface implemented by Ceph, MinIO and every major object store, so the evidence store can move without changing its guarantee. The trail records facts (commits, API events, certificates) rather than tool-specific logs.

> **Lesson.** Keep evidence where its immutability is guaranteed by the storage, and treat every search tool as a copy of it.

### Observability and cost

*What every hop reports, how a failure is attributed, how drift is seen, and how the mesh's own cost is kept inside a number.*

#### ADR-28 · Prometheus per cluster with Thanos for global query and history; labels pruned at the proxy

**Status:** Accepted  ·  **Shown on views:** 15, 25, 07

*How are per-hop metrics collected from 40,000 proxies without the metrics system becoming the most expensive part of the mesh?*

**Context.** The requirement asks for rate, errors and duration for every hop by source identity, destination service, subset and response class, retained 15 days raw and 13 months rolled up. Envoy emits thousands of series per proxy by default, and per-pod labels multiply them by 40,000. Rollout analysis and wave gates read these metrics, so the system must answer within seconds during an incident.

**Decision.** Each cluster runs Prometheus in HA pairs scraping its proxies. Istio's Telemetry API reduces standard metrics to the labels the requirement names, and Envoy's own stats are limited to an allowlist. Recording rules aggregate to service-to-service series; per-pod labels are dropped at scrape. Thanos sidecars upload blocks to Ceph RGW; Thanos compactor downsamples to 5-minute and 1-hour resolution for 13 months; Thanos Query at each site serves the site, and a global query layer serves cross-site views.

**How it works on-premise.** Prometheus runs on the platform node pool with local SSDs for the 2-hour head block. Thanos stores and compactors run in each site's management cluster against a site-local Ceph RGW bucket. Rollout analysis and gates query the site-local Thanos, never the global layer, so a site partition does not blind them. Alertmanager runs per site with a cross-site mesh for deduplication.

| Option | Verdict | Reasoning |
|---|---|---|
| Prometheus per cluster with Thanos | Chosen | Collection stays local to the failure domain, history is cheap on object storage, and the query path for gates is site-local. |
| VictoriaMetrics cluster | Rejected | More efficient storage and simpler scaling. Clustering and some enterprise features differ between editions, and the team already runs Prometheus. |
| Grafana Mimir | Rejected | Horizontally scalable and multi-tenant. More moving parts than a platform team of this size needs for 20 clusters. |

**What it buys**

- Metrics collection survives a site partition and a telemetry backend outage
- Thirteen months of history cost object storage, not SSD
- Label pruning keeps series count proportional to service pairs, not pods

**What it costs**

- Per-pod debugging needs a temporary, scoped label override
- Thanos adds components (sidecar, store, compactor, query) to operate
- Cross-site queries are slower and fail during partitions

**Choose differently when.** If per-series cost becomes the limiting factor at the design ceiling, move storage to VictoriaMetrics behind the same Prometheus query API; dashboards and gates do not change.

**Why it holds up over time.** PromQL and the Prometheus exposition format are the most stable interfaces in observability, and OpenTelemetry metrics map onto them. Choosing the label set deliberately, in the Telemetry API, is what keeps cost stable as the fleet grows, whichever backend stores it.

> **Lesson.** Decide which labels a metric may carry before you decide where to store it; cardinality, not storage, sets the cost of mesh telemetry.

#### ADR-29 · Access logs to ClickHouse and traces to Tempo, sampled per namespace, never blocking a request

**Status:** Accepted  ·  **Shown on views:** 15, 25, 26

*How are per-hop access logs and spans kept useful without being the runaway cost of the mesh?*

**Context.** At 400,000 requests a second through two proxies each, full access logging is about 70 billion lines a day. The requirement asks for access logs with peer identity, route, subset, retry count and terminating condition for 30 days, a span per proxy hop for 7 days, sampling and cardinality configurable per namespace with a declared default, no payloads or authorisation header values by default, and bounded, drop-oldest buffering that never blocks a request.

**Decision.** Envoy sends access logs over gRPC to a node-local OpenTelemetry Collector, which forwards to ClickHouse. Default capture is every denial and every 4xx or 5xx, plus 1% of successes; a namespace raises it with a reviewed change with an expiry date. Spans are emitted per hop with head-based sampling at 1% by default, always sampled when the caller has already sampled, and stored in Grafana Tempo on Ceph RGW. Collectors use bounded queues that drop the oldest data and count drops.

**How it works on-premise.** ClickHouse runs as a replicated cluster per site with a table partitioned by day, a 30-day TTL, and a projection on peer identity for incident queries. Header values are excluded by the access-log format; the log carries header names only where declared. Tempo uses Ceph RGW for blocks with a 7-day retention. Collector drop counters are scraped by Prometheus and alert above 0.1% sustained.

| Option | Verdict | Reasoning |
|---|---|---|
| ClickHouse for logs, Tempo for traces, per-namespace sampling | Chosen | Columnar storage suits high-volume structured logs; object-storage traces are cheap; the default keeps every error. |
| OpenSearch for logs | Rejected | Full-text search many teams know. Indexing cost at this volume is several times ClickHouse's for structured fields. |
| Jaeger for traces | Rejected | CNCF graduated and widely used. Needs a separate storage cluster where Tempo uses the object store already run for Thanos. |
| Log everything, sample nothing | Rejected | Never misses a request. Makes telemetry the largest line in the mesh budget. |

**What it buys**

- Every failure and denial is logged in full; successes are sampled
- A slow or full telemetry backend never slows a request
- Privacy defaults are enforced by the log format, not by convention

**What it costs**

- A rare successful call may not be in the logs, which weakens the impact replay in ADR-12
- Two storage systems for two signal types
- Teams that need full success logging pay for it visibly and temporarily

**Choose differently when.** For a low-volume, highly regulated service (payments settlement), log every request unsampled in its own namespace policy with a longer retention, and account for it in that team's budget.

**Why it holds up over time.** OpenTelemetry is the collection standard, so the backends can change without touching 40,000 proxies. The sampling policy is data per namespace, which lets cost follow value as traffic grows rather than as a global compromise.

> **Lesson.** Keep every error and a sample of success, make exceptions explicit and temporary, and never let the record of a request slow the request.

#### ADR-30 · Every failure signal carries its origin: application, mesh policy or configuration, or network

**Status:** Accepted  ·  **Shown on views:** 15, 20, 31

*When a call fails, how does anyone know whether to page the service team, the mesh team or the network team?*

**Context.** A 503 can mean the application returned it, the callee's proxy shed load, no healthy upstream existed, an authorisation rule denied it, the route was missing, or a TLS handshake failed on a trust-bundle mismatch. Without distinguishing these, every mesh incident starts with a debate about whose fault it is, and the 99.99% mesh-attributable availability target cannot be measured. The requirement asks for that distinction in every failure signal.

**Decision.** Every request metric and access log carries a failure_origin label with four values (none, application, mesh, network) derived from Envoy's response flags and response code details. Mesh covers authorisation denial, no route, overflow, rate limit, and configuration or certificate errors; network covers connection failure, reset and upstream timeout without a response; application covers responses the upstream application produced. TLS failures are further split into trust-bundle skew and certificate validation, so skew is never reported as an authorisation problem.

**How it works on-premise.** The label is added through Istio's Telemetry API with a CEL expression over response.flags and response.code_details, and the mapping table lives in the platform repository with tests. The mesh availability SLO is computed from failure_origin="mesh" over all requests. Pages route by origin: mesh to the platform on-call, application to the owning team, network to the network on-call with the pair of localities.

| Option | Verdict | Reasoning |
|---|---|---|
| Origin label from response flags on every signal | Chosen | Makes the mesh's own availability measurable and routes incidents to the right team from the first alert. |
| Status codes and dashboards only | Rejected | No extra labels. Every 503 looks the same, and the mesh SLO becomes a matter of opinion. |
| Distinct status codes for mesh failures | Rejected | Visible to applications. Changes response semantics clients depend on and breaks retries keyed on 503. |

**What it buys**

- The 0.01% mesh-attributable failure target is a query, not an argument
- On-call routing is correct at the first page
- Trust-bundle skew during a root rotation is visible as skew

**What it costs**

- The mapping from flags to origin must be maintained as Envoy adds flags
- One more label on request metrics (four values, so a bounded cost)
- Some ambiguous cases, such as an upstream timeout caused by a slow application, are classified by rule and can be wrong

**Choose differently when.** A small mesh run by the same team that owns the services can skip the label and debug from access logs directly, because all pages go to the same people anyway.

**Why it holds up over time.** Every proxy exposes some form of termination reason; the stable part is the four-value classification and the SLO computed from it. If the proxy changes, only the mapping table does.

> **Lesson.** A platform on the request path must be able to say, per failure, whether it caused it, or it will be blamed for everything and measured on nothing.

#### ADR-31 · Drift is observed per proxy, not inferred from sync status

**Status:** Accepted  ·  **Shown on views:** 13, 14, 25

*How does anyone know that the configuration a proxy runs is the configuration that was intended?*

**Context.** Argo CD reporting Synced means objects are in etcd. istiod reporting a push means it sent something. Neither proves a proxy acknowledged it, and delta distribution adds a failure mode where a proxy diverges from intent with no error anywhere. The requirement asks that each proxy's effective configuration and version be exposed, so that drift is observed rather than inferred, and that any effective configuration be traceable to the intent version that produced it.

**Decision.** Every applied mesh object carries the commit sha that produced it. A drift exporter in each cluster reads istiod's per-proxy sync state (last sent and acknowledged nonce per resource type, proxy version) and the shas of the objects in each proxy's scope, and exports per proxy: the newest acknowledged sha, seconds behind intent, NACK count and proxy version. A proxy more than 60 seconds behind is forced to resubscribe; wave gates and rollout analysis read the same series.

**How it works on-premise.** The exporter is a small Go service using istiod's debug sync endpoint through an authenticated port, with the same data istioctl proxy-status shows, exported as Prometheus metrics with the proxy identity as a label only for proxies that are behind (to bound cardinality). Grafana shows drift per cluster, namespace and change sha.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-proxy acknowledged sha, exported continuously | Chosen | Effective state is a metric; drift is a query; gates act on it automatically. |
| Argo CD sync status | Rejected | Already available. Proves objects reached the API server, not that proxies run them. |
| istioctl proxy-status on demand | Rejected | Accurate. Only when someone thinks to run it, which is after users noticed. |

**What it buys**

- A change is known to be live on a given share of proxies, with numbers
- Silent divergence from delta distribution is detected within a minute
- An auditor can trace a proxy's effective configuration to a signed commit

**What it costs**

- A platform component that depends on istiod debug interfaces, which can change between releases
- Forced resubscription adds control-plane load if many proxies lag at once
- Cardinality control means per-proxy detail exists only for proxies that are behind

**Choose differently when.** If Istio exposes acknowledged configuration version per proxy as a first-class metric in a future release, replace the exporter with it and keep the gates and dashboards unchanged.

**Why it holds up over time.** The principle is to report effective state from the component that enforces it and to stamp intent with a version. Any xDS control plane has nonces and versions, so the exporter's contract survives a change of mesh.

> **Lesson.** Never treat 'I sent it' as 'it is running'; stamp intent with a version and make the enforcing component report the version it holds.

#### ADR-32 · The mesh's cost is a declared budget, measured monthly and attributed to teams with OpenCost

**Status:** Accepted  ·  **Shown on views:** 26, 25, 27

*How is the cost of the mesh kept inside a number instead of being discovered at renewal?*

**Context.** The mesh's cost is spread across every pod, every request and every byte of telemetry, which makes it easy to under-count and hard to reverse. Planning puts the idle floor of 40,000 proxies at about 400 vCPU, five times the per-request cost, and memory up to 4.8 TB. The requirement sets the mesh at no more than 8% of fleet CPU and 10% of fleet memory, reported monthly against actuals, attributed to owning teams, including telemetry volume and cross-zone traffic caused by routing.

**Decision.** OpenCost attributes the CPU and memory of every istio-proxy container, gateway and control-plane pod to namespaces and, through namespace labels, to teams. Telemetry volume per namespace comes from collector and ClickHouse metrics, and cross-zone and cross-cluster bytes from locality labels on request metrics. A monthly report compares the mesh's share of fleet CPU and memory with the budget and lists each team's share. Crossing 80% of either budget opens a review of scoping, sampling and the per-node L4 option.

**How it works on-premise.** Node costs are an internal rate per vCPU and GB of memory supplied by finance, loaded into OpenCost's custom pricing. Cross-site link cost is an internal rate per GB. Sidecar resource requests are set from measured usage percentiles per workload size class, not from defaults, and reviewed each quarter with the upgrade.

| Option | Verdict | Reasoning |
|---|---|---|
| Declared budget, OpenCost attribution, monthly report per team | Chosen | The team that meshes a service sees its cost; the platform sees its trend against a number. |
| Platform absorbs the cost centrally | Rejected | Encourages adoption. Nobody who can change sampling or scoping sees the bill. |
| Estimate once at design time | Rejected | Cheap. The idle floor and telemetry volume change with every release and every new service. |

**What it buys**

- Cost regressions from a mesh upgrade or a sampling change are visible within a month
- Teams have a reason to declare narrow dependencies and keep sampling at default
- The per-node L4 decision in ADR-03 is made on measured numbers

**What it costs**

- Attribution depends on accurate namespace-to-team labels
- Internal rates are estimates finance must maintain
- Chargeback can discourage teams from meshing unless the first year is shown but not charged

**Choose differently when.** During initial adoption, show costs without charging them for a year; charging from day one makes the cheapest option not meshing at all.

**Why it holds up over time.** A declared budget with a measured actual is the only cost control that survives changes in hardware prices, proxy efficiency and fleet size, because it compares ratios rather than absolute numbers. OpenCost reads Kubernetes resource data that any scheduler-level tool can provide.

> **Lesson.** A shared platform's cost must be a number someone owns and every tenant can see, or it grows until it is noticed at the worst time.

## Every package used, in one table

Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| Istio | Service mesh, CNCF graduated | Control plane (istiod), sidecar injection, mTLS, authorisation, traffic policy, multi-cluster, revision upgrades | Linkerd, Cilium service mesh, Kuma |
| Envoy | L4/L7 proxy, CNCF graduated | Sidecar, east-west gateway and egress gateway data plane | linkerd2-proxy |
| Kubernetes Gateway API | Kubernetes routing API with a mesh profile | HTTPRoute and GRPCRoute for weights, cohorts, mirroring and timeouts | Istio VirtualService |
| SPIRE | SPIFFE reference implementation, CNCF graduated | Node and workload attestation, SVID issuance, 7-day in-cluster CA, issuance audit | istiod CA, cert-manager with istio-csr |
| spire-controller-manager and SPIFFE CSI driver | SPIRE's Kubernetes integrations | Registration entries from ClusterSPIFFEID templates; Workload API socket into pods | Hand-written entries; hostPath mounts |
| OpenBao | Secrets and PKI server, Linux Foundation fork of Vault | Per-cluster intermediate PKI mounts; signing key for bundles | Vault (BSL), step-ca |
| OpenSSL with a PKCS#11 provider | Cryptographic toolkit and HSM interface | Offline root ceremony on an air-gapped workstation | Vendor HSM tooling |
| Kubernetes | Container orchestrator | Workload platform, native sidecars, ValidatingAdmissionPolicy, audit log | Nomad |
| Cilium | eBPF CNI, CNCF graduated | Pod networking, BGP route and VIP announcement, L3/L4 policy as second enforcement plane | Calico with MetalLB |
| Argo CD | GitOps reconciler, CNCF graduated | Per-site delivery of mesh intent in cluster waves, with kind allowlists per project | Flux |
| Argo Rollouts | Progressive delivery controller | Sole writer of canary weights; analysis on Thanos; automatic abort | Flagger |
| Forgejo | Self-hosted Git service | Three intent repositories with signed commits and protected branches; mirrors per site | GitLab CE, Gitea |
| Harbor | OCI registry, CNCF graduated | Mirrored mesh images and signed configuration bundles for one-step revert | Zot |
| cosign and ORAS | Artefact signing and OCI packaging tools | Sign and package rendered bundles; verified at admission | Notation |
| OPA Gatekeeper | Policy admission controller for Kubernetes, part of OPA (CNCF graduated) | Context-dependent admission rules; the same Rego as CI via conftest | Kyverno |
| Open Policy Agent | General policy engine, CNCF graduated | Phase 3 external authorisation sidecar for opted-in services; CI policy checks | Custom authz service |
| PostgreSQL with CloudNativePG | Relational database and its Kubernetes operator | SPIRE datastore per cluster; Forgejo database | MySQL |
| Prometheus | Metrics system, CNCF graduated | Per-cluster scraping of proxy and control-plane metrics; recording rules | VictoriaMetrics agent |
| Thanos | Long-term storage and global query for Prometheus | 13-month downsampled history on Ceph; site-local query for gates and analysis | VictoriaMetrics, Grafana Mimir |
| OpenTelemetry Collector | Vendor-neutral telemetry pipeline | Node-local bounded buffering of access logs, spans and audit events | Fluent Bit, Vector |
| ClickHouse | Columnar analytical database | Access logs (30 days) and issuance records (13 months); impact replay queries | OpenSearch |
| Grafana Tempo | Trace store on object storage | Per-hop spans, 7 days | Jaeger |
| Grafana and Alertmanager | Dashboards and alert routing | Per-service views, drift and cost dashboards; pages routed by failure origin | Perses |
| OpenCost | Kubernetes cost allocation | Proxy and gateway cost attributed to teams against the mesh budget | Spreadsheet from kube-state-metrics |
| Ceph RGW | S3-compatible object storage | Thanos and Tempo blocks; Object Lock bucket for the audit trail | MinIO |
| chrony and node-problem-detector | Time synchronisation daemon and node condition reporter | Hold clock skew within ±60 s and taint nodes that drift beyond it | ntpd, custom DaemonSet |