Service Mesh Platform

Architecture Views

31 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

A service mesh for 2,000 services and 40,000 proxies across 20 Kubernetes clusters in three data centres the organisation owns. It is built from open-source software: Istio with Envoy sidecars for the data plane and configuration compiler, SPIRE for attested workload identity under an offline HSM root with per-cluster intermediates in OpenBao, Cilium for pod networking and a second enforcement plane, Argo CD and Argo Rollouts for intent and weights, and Prometheus, Thanos, Tempo and ClickHouse for telemetry. Read the set in order. Acts 1 and 2 fix the boundary and the people who change it. Act 3 shows the layers, one cluster, the trust hierarchy and what happens inside a single call. Act 4 separates five stores by what losing each would cost. Act 5 walks issuance, a root rotation, a failed canary, a cross-cluster failover, a retry storm and a total control-plane outage. Acts 6 and 7 cover how it is deployed, changed, upgraded, watched and paid for, then what could go wrong. One boundary governs every page: the control plane compiles configuration ahead of time and is never asked anything while a request is being forwarded.

1 · Context and scope

What the mesh owns on the east-west path, who changes it, and what it stands on without owning.
01 Traffic it carries Meshed workloads 2,000 services North-south ingress edge gateways People who change what it does Service team Release engineer Security engineer Mesh SRE What it stands on and reaches Kubernetes clusters scheduling · endpoints Offline root CA HSM · ceremony External APIs declared egress only Delivery pipeline CI · image builds Service Mesh Platform 20 clusters · 40k proxies Records and reporting Git service intent repositories Team cost owners monthly report east-west calls admitted routes · dependencies rollouts · revert authz · identity upgrades · drift pods · endpoints signs intermediates controlled egress rollout intent declared intent mesh cost Service Mesh Platform — System Context External / third party Person or role Security / platform synchronous batch v 1.0 · owner Platform Networking Architecture · date 2026-09 System Context The traffic the mesh carries, the people who change its behaviour, and the platform and authorities it depends on without owning. HTML page SVG draw.io
02 Declare Intent repositories Forgejo · 3 repos Argo Rollouts owns canary weights Admit Policy checks OPA · impact replay Admission guards Gatekeeper · VAP Compile istiod scoped · delta xDS SPIRE server attested SVIDs Enforce locally Envoy sidecar mTLS · authz · routes East-west gateway SNI passthrough Observe OTel collector bounded buffers Drift exporter intent sha per proxy effective weights High-Level Architecture — Intent In, Local Enforcement Out Data store Application we own Decision point Security / platform Interface / broker event / async Nothing left of the data plane is consulted while a request is forwarded. Every arrow into it is a push that can stop without stopping traffic. v 1.0 · owner Platform Networking Architecture · date 2026-09 High-Level Architecture Five stages from declared intent to local enforcement and back, with the one rule that shapes all of them. HTML page SVG draw.io

2 · People and journeys

Who the mesh is for, and the three moments where it either earns its place on every request path or becomes the incident.
03 Teams that own services Service team ≈ 250 teams Goal — Get mTLS, retries and a canary without writing any of it into my service. Core journeys Move a service to strict mTLS Declare a new dependency Read what my mesh costs Release engineer ≈ 900 releases a week Goal — Ship to 1% of traffic, and get back in one step if it is wrong. Core journeys Canary a release at 1% Mirror traffic to a new version Abort a rollout Platform and security Mesh SRE on call for 20 clusters Goal — Upgrade 40,000 proxies without anyone noticing, and see drift before users do. Core journeys Upgrade the proxy fleet in waves Ride out a control-plane outage Explain one failed call Security engineer owns authz and PKI Goal — Know who called what, and make a stolen identity useless within minutes. Core journeys Contain a compromised workload Dry-run an authorisation policy Rehearse a root rotation Machines and reviewers Rollout controller Argo Rollouts Goal — Read the weights the mesh actually applied, not the ones I asked for. Core journeys Step weights on analysis Abort on SLO breach Auditor 13-month lookback Goal — Show me every certificate issued and every policy change, with who did it. Core journeys Query issuance records Trace a change to its author Actors — Who the Mesh Is For and What They Get to Do Person or role Journey / task Security / platform External / third party v 1.0 · owner Platform Networking Architecture · date 2026-09 Actors and Their Core Journeys Six parties who change or depend on the mesh, each with the one thing they need from it, and the journeys that carry the value. HTML page SVG draw.io
04 Service team payments-ledger Goal — Refuse plaintext without breaking a caller Trigger — Audit finding: 3% of calls still plaintext Done when — Strict mode, no legitimate call denied 1 · Find callers 2 · Mesh them 3 · Write allow 4 · Dry run 5 · Go strict ◆ moment of truth 6 · Hold What they do Opens plaintext report Meshes two batch jobs Writes allow by identity Runs policy in dry-run Merges the strict flip Watches denials 7 days System behind it Plaintext report Injection policy Security intent repo Shadow rules Config waves Denial dashboard How it feels Confident Unsure Anxious Where it hurts Who are the 3%? Legacy job resists Which principals? Feels one-way What the platform gives Callers named Opt-out per workload Survives rescheduling Would-deny list, live Impact gate at 0.5% Revert in one step Journey — A Service Team Refuses Plaintext Without Breaking a Caller v 1.0 · owner Platform Networking Architecture · date 2026-09 Journey — Move a Service to Strict mTLS The payments-ledger team closes an audit finding. The low points are finding the callers they did not know about, and the moment plaintext is refused. HTML page SVG draw.io
05 Release engineer checkout v2.14 Goal — Find a bad build on 1% of traffic Trigger — Merged change to the pricing path Done when — Promoted, or reverted inside 60 s 1 · Declare 2 · Mirror 3 · Cohort 4 · 1% live ◆ moment of truth 5 · Widen 6 · Promote What they do Opens the rollout Mirrors reads to v2.14 Routes staff by header Starts general 1% Steps 5 → 25 → 50% Promotes to 100% System behind it Argo Rollouts Mirror route Header match Weighted route Analysis on Thanos Stable subset retired How it feels Confident Unsure Anxious Where it hurts Mirror writes twice? Did 1% really apply? p99 up 40 ms at 25% What the platform gives One owner of weights Only GET mirrored Named cohort first Applied share read back Auto abort ≤ 60 s Every step revertible Journey — A Release Engineer Canaries checkout v2.14 at 1% v 1.0 · owner Platform Networking Architecture · date 2026-09 Journey — Canary a Release at 1% A release engineer ships checkout v2.14. The anxious moment is the first 1% of real traffic, and the question is whether the mesh did what it was told. HTML page SVG draw.io
06 Security engineer on call Goal — Make a stolen workload identity useless Trigger — report-gen calling payments, never seen before Done when — Cut off in minutes, dead in 24 h, audit complete 1 · Detect 2 · Attribute 3 · Cut off ◆ moment of truth 4 · Evict 5 · Expire 6 · Report What they do Gets undeclared-call alert Queries access logs Merges emergency deny Bans node · kills pod Waits out the SVID Pulls issuance record System behind it Alertmanager Access logs Deny policy · Cilium SPIRE agent ban 24 h lifetime Issuance records How it feels Calm Tense Alarmed Where it hurts Pod IP already reused Deny rides a config push Stolen cert still valid What the platform gives Caller named by identity Peer identity per hop L3 block, second plane Key proves one identity Useless after ≤ 24 h Attestation per cert Journey — A Security Engineer Makes a Stolen Identity Useless v 1.0 · owner Platform Networking Architecture · date 2026-09 Journey — Contain a Compromised Workload A report-generation pod starts calling the payments service. The engineer has to make a stolen identity useless while the certificate is still valid. HTML page SVG draw.io

3 · Structure

The layers, one cluster's containers, the trust hierarchy, every interface, and the local decisions inside one meshed call.
07 Intent and governance Intent repositories Forgejo · signed Policy checks OPA · impact replay Argo CD one per site · waves Argo Rollouts weights owner Control plane Kubernetes API etcd · per cluster Admission guards Gatekeeper · VAP istiod compile · delta xDS Identity plane Offline root HSM · 10 years OpenBao PKI cluster intermediates SPIRE server 7-day signing CA SPIRE agent per node · attests Data plane Cilium L3/L4 policy · BGP Egress gateway declared hosts East-west gateway SNI passthrough Envoy sidecar native sidecar Observability and cost OpenCost per-team report Thanos 15 d · 13 mo Tempo traces · 7 d OTel collector node · bounded ClickHouse access · issuance apply delta xDS SVID · SDS telemetry Layered Architecture Data store Decision point Application we own Security / platform Interface / broker synchronous event / async Each layer depends only on the layers above it for change, and on nothing above it for forwarding a request. v 1.0 · owner Platform Networking Architecture · date 2026-09 Layered Architecture Five layers, from declared intent to telemetry, with the packages that implement each. HTML page SVG draw.io
08 Cluster c07 · site A · Kubernetes Platform node pool Control plane Kubernetes API 3 × etcd istiod 3 replicas · HPA Admission guards Gatekeeper · VAP Identity SPIRE server 3 replicas SPIRE datastore CloudNativePG Workload node Pod · checkout checkout app unaware of mesh Envoy sidecar native sidecar Node agents · DaemonSets SPIRE agent Workload API OTel collector bounded queue Cilium agent eBPF · policy Gateway node pool East-west gateway BGP VIP · passthrough Egress gateway declared hosts OpenBao PKI site A Telemetry stores Thanos · Tempo · ClickHouse Remote clusters their east-west gateways watch delta xDS SVIDs localhost CA renewal OTLP mTLS · SNI Container Architecture — One Workload Cluster Security / platform Application we own Data store Interface / broker External / third party synchronous event / async Omitted for legibility: Argo CD applying to the API (view 14), sidecar to agent over SDS (view 16), sidecar to collector (view 15). v 1.0 · owner Platform Networking Architecture · date 2026-09 Container Architecture — One Workload Cluster What runs inside one of the twenty clusters, which pool each piece lives in, and the four things it reaches outside the cluster. HTML page SVG draw.io
09 Offline · 10 years Mesh root HSM · air-gapped Site OpenBao · 1 year Intermediate c07 name-constrained Intermediate c12 name-constrained In cluster · 7 days SPIRE CA c07 renews at 3.5 d SPIRE CA c12 renews at 3.5 d Per pod · 24 hours SVID checkout rotated at 12 h SVID ledger rotated at 12 h Every proxy Trust bundle roots only · sent ahead ceremony ceremony signs signs signs signs verified against Trust Hierarchy — Four Lifetimes, Each Shorter Than the One Above Security / platform Application we own Data store batch synchronous Intermediate c07 can sign only spiffe://c07.mesh.corp identities. A proxy trusts roots, never an intermediate. v 1.0 · owner Security Architecture · date 2026-09 Trust Hierarchy Four certificate lifetimes, each much shorter than the one that signs it, and a trust bundle that carries roots only. HTML page SVG draw.io
10 What changes it Intent repos · Argo CD Argo Rollouts Kubernetes API Platform Service Mesh Platform compile · issue · enforce What it serves and reaches Workloads · peer clusters Telemetry stores OpenBao · root ceremony GitOps apply weight steps watch mTLS · xDS OTLP CA renewal Integration Surface — Every Interface Data store Application we own Security / platform External / third party synchronous event / async Only mTLS between proxies is on a request path. External APIs sit behind the egress gateway on view 29. v 1.0 · owner Platform Networking Architecture · date 2026-09 Integration Surface Every interface the platform exposes or consumes, grouped by whether it changes the mesh or is served by it. HTML page SVG draw.io
11 Caller pod checkout app forwards trace · deadline Outbound proxy Route match subset · weight Timeout · retry budget GET only · 20% Outlier · limits locality first On the wire mTLS SVID · TLS 1.3 Inbound proxy Identity verify chain + expected SAN Authorisation in-proxy · 0.2 ms Concurrency limit shed, never queue Callee pod ledger app sees caller identity Anatomy of One Meshed Call — Every Decision Reads Local State Application we own Interface / broker Decision point Security / platform Three local inputs decide everything here: last-good config, the trust bundle and an unexpired SVID. None is fetched per request. v 1.0 · owner Platform Networking Architecture · date 2026-09 Anatomy of One Meshed Call The eight decisions two proxies make between checkout and ledger, and the three local inputs every one of them reads. HTML page SVG draw.io

4 · Data

Five stores with five recovery obligations, the intent-and-evidence model, and the two pipelines that move configuration out and telemetry back.
12 Mesh state, by what losing it costs Intent · source of truth · RPO 0 Intent repositories Forgejo · signed commits Applied objects etcd · per cluster Identity · RPO 0 OpenBao Raft intermediates SPIRE datastore entries · agents Issuance records ClickHouse · 13 mo Derived · rebuildable xDS snapshots istiod memory · ≤ 5 min Rendered bundles Harbor OCI · cosign Discovery · current only EndpointSlices per cluster Remote endpoints declared exports only Telemetry · loss-tolerant Thanos 15 d raw · 13 mo Tempo traces · 7 d Access logs ClickHouse · 30 d Five Stores, Five Recovery Obligations Only the first two columns need backups that are restored in drills. The third is rebuilt by a restart, the fourth by the clusters themselves. v 1.0 · owner Platform Networking Architecture · date 2026-09 Five Stores, Five Recovery Obligations Where each kind of state lives, what it costs to lose it, and which stores need backups that are actually restored in drills. HTML page SVG draw.io
13 intent_version commit_sha PK repo routes|security|identity author signed_by bundle_digest merged_at change_record change_id PK commit_sha FK actor before_digest after_digest applied_at per cluster service service_id PK namespace owner_team mtls_mode permissive|strict sampling_rate subset subset_id PK service_id FK selector version, region backing_service dependency caller_id FK -> service callee_id FK -> service declared_in FK -> intent_version route route_id PK service_id FK match header, path, method timeout_ms NOT NULL retry_on, attempts commit_sha FK route_backend route_id FK subset_id FK weight 0-100 mirror_percent authz_policy policy_id PK service_id FK action allow|deny mode enforce|dry_run principals commit_sha FK workload_identity spiffe_id PK trust_domain cluster namespace service_account issuance_record serial PK spiffe_id FK node_attestation workload_selectors issuer_ca not_before, not_after proxy_status proxy_id PK spiffe_id FK commit_sha FK effective xds_nonce, acked_at proxy_version 1 : N 1 : N 1 : N 1 : N N : M 1 : N 1 : N 1 : N Intent and Evidence — Data Model proxy_status.commit_sha is observed, route.commit_sha is intended; drift is the rows where they differ. v 1.0 · owner Platform Networking Architecture · date 2026-09 Intent and Evidence — Data Model The entities that declared intent, identity and observed state share, and the one join that makes drift a query instead of a guess. HTML page SVG draw.io
14 Author Intent change one of three repos Check Schema + lint timeout required Impact replay 24 h of access logs Package Rendered bundle OCI · cosign Apply Argo CD wave canary cluster first Admission guards second line Compile istiod scoped per workload Deliver Envoy sidecar delta · ACK or NACK Verify Drift exporter sha per proxy halt wave Configuration Distribution — From Merge to Acknowledged Data store Decision point Application we own Security / platform Interface / broker failure / alternate A push is finished when proxies acknowledge it and the drift exporter sees the new sha, not when Argo CD reports Synced. v 1.0 · owner Platform Networking Architecture · date 2026-09 Configuration Distribution Pipeline The seven steps from a merged change to a configuration every affected proxy has acknowledged, and the check that closes the loop. HTML page SVG draw.io
15 Emit Envoy sidecar stats · spans · ALS Node OTel collector drop-oldest · counted Shape Sampling policy per namespace Failure origin tag app · mesh · network Keep Prometheus + Thanos 15 d · 13 mo Tempo 7 d Access logs ClickHouse · 30 d Use Grafana per-service views Rollout analysis Argo Rollouts OpenCost team attribution Telemetry Pipeline — Never on the Request Path Interface / broker Security / platform Decision point Application we own Data store Default capture: all denials and 5xx, 1% of successes, no header values and no payloads. Namespaces raise it by a reviewed change. v 1.0 · owner Platform Networking Architecture · date 2026-09 Telemetry Pipeline How metrics, spans and access logs leave the proxy, where they are shaped and sampled, and who reads them. HTML page SVG draw.io

5 · Runtime

Issuance and rotation, a root rotation, a canary that fails, a cross-cluster failover, a retry storm contained, and a total control-plane outage hour by hour.
16 Envoy sidecar SPIRE agent kubelet SPIRE server OpenBao PKI Peer proxy 1. SDS: fetch SVID (Workload API) 2. which pod owns this PID? 3. namespace · service account · labels 4. CSR for attested selectors 5. match entry · write issuance record 6. SVID 24 h + trust bundle 7. push SVID and bundle 8. mTLS handshake 9. t + 12 h: renew 10. rotated SVID · no restart 11. renew 7-day CA at half-life 12. CA signed by c07 intermediate Certificate Issuance and Rotation — Attested, Never Asserted The pod never names itself. Its identity comes from what the kubelet reports about the process on a node the server has already attested. v 1.0 · owner Security Architecture · date 2026-09 Certificate Issuance and Rotation How a sidecar gets an identity it did not choose, renews it with no restart, and how the cluster CA renews itself independently. HTML page SVG draw.io
17 T − 30 d · add Overlap T0 · switch T0 → 24 h T + 30 d · retire Trust bundle Old + new root Reaches every proxy Old root removed Intermediates Re-signed by new root SPIRE CAs chain to new Workload SVIDs New issuance, new chain Old-chain SVIDs expire Gate Bundle coverage 100% Skewed handshakes = 0 Hold if any proxy lags Old-chain count → 0 Removal PR, two keys Root Rotation — Additive, With a 30-Day Overlap Every step can be reversed until the last, and the last is taken only after the overlap has been measured, not assumed. v 1.0 · owner Security Architecture · date 2026-09 Root Rotation Replacing the mesh root without a single handshake failing, by making every step additive and measuring before taking the next. HTML page SVG draw.io
18 Release engineer Argo Rollouts Kubernetes API istiod Caller proxies Thanos 1. start v2.14 at 1% 2. route weights 99 / 1 3. watch event 4. delta xDS: one route 5. ACK 6. applied canary share? error ratio? 7. 0.98% · 5xx 2.3× stable 8. analysis failed: abort 9. route weights 100 / 0 10. delta xDS: one route 11. aborted · 41 s after breach A Canary That Fails — Automatic Rollback in 41 Seconds The rollout checks the share the mesh reports, not the weight it wrote. Only Argo Rollouts may write this route; a human aborts the rollout, not the route. v 1.0 · owner Platform Networking Architecture · date 2026-09 A Canary That Fails checkout v2.14 is started at 1%, its error ratio breaches, and the rollout reverts itself without a human touching a route. HTML page SVG draw.io
19 checkout · c07 checkout proxy · c07 ledger pods · c07 East-west gateway · c12 ledger proxy · c12 ledger app · c12 1. GET ledger/balance 2. try local zone 3. 5 consecutive 503s 4. eject · local healthy below 70% 5. TLS · SNI names ledger subset 6. bytes forwarded, TLS untouched 7. peer is c07/checkout · allowed 8. request 9. 200 10. 200 · spillover counted Cross-Cluster Failover — The Caller's Identity Arrives Intact The gateway reads SNI and nothing else. It holds no workload identity, so it cannot be the caller. v 1.0 · owner Platform Networking Architecture · date 2026-09 Cross-Cluster Failover The local ledger pods fail, the caller spills to cluster c12 through an east-west gateway, and ledger still sees checkout as the caller. HTML page SVG draw.io
20 Dependency slows Retries begin Budget reached Shedding Recovery Caller proxies p99 280 ms → 2 s Retry 503 · GET only Budget 20% spent Fast 503 to app Budget refills Callee proxies Pending grows Outlier ejects 2 pods Concurrency at limit Overflow shed Ejections expire Deadline Caller allows 800 ms 250 ms per try No retry past deadline Signals Latency alert Retry ratio 0.2 retry_overflow Origin: mesh limit Budget refills A Slow Dependency, and Why It Does Not Become an Outage Without the budget, three attempts per call triples load on the one service already failing. With it, load rises by at most a fifth. v 1.0 · owner Platform Networking Architecture · date 2026-09 A Slow Dependency, Contained A dependency's latency rises tenfold. Four mechanisms stop three retry attempts per call from tripling load on the one service already failing. HTML page SVG draw.io
21 0 – 12 h 12 – 24 h Beyond 24 h When it returns Existing traffic Flows on last config Flows · expiry paging Fails closed per pod Renew · reconnect Certificate renewal Not yet due Renewals fail, retried Emergency issuance Backlog issued ≤ 20 s Routes and denies Frozen at last push Frozen Frozen Push · drift clears New pods Refused in strict ns Refused in strict ns Refused in strict ns Config · SVID · ready Crashed proxy Pod unready · drained Pod unready · drained Pod unready · drained Restart fetches config Emergency deny Cilium L3 policy only Cilium L3 policy only Cilium L3 policy only Mesh deny lands Total Control-Plane Outage — A Freeze, Then a Deadline istiod, SPIRE server and OpenBao all down together. 12 hours is the promise; 24 hours is the hard edge set by certificate lifetime. v 1.0 · owner Platform Networking Architecture · date 2026-09 Total Control-Plane Outage istiod, SPIRE server and OpenBao are all down in one cluster. Traffic freezes on last-known-good for 12 hours, pages before 24, and fails closed after. HTML page SVG draw.io

6 · Operations

Where it runs, how a change and an upgrade reach 40,000 proxies, what is watched, where the money goes, the build order and the release loop.
22 Site A · three rooms Seven workload clusters, each spread across rooms Room 1 istiod replica 1 SPIRE server replica 1 Workers sidecars · agents Room 2 istiod replica 2 SPIRE server replica 2 Workers sidecars · agents Room 3 istiod replica 3 SPIRE server replica 3 East-west gateways BGP VIP · ECMP Site services Argo CD site A clusters only OpenBao PKI 5 voters · 3 rooms Forgejo primary Harbor images · bundles Thanos · Tempo · ClickHouse site stores Sites B and C · same shape Thirteen clusters own istiod · SPIRE Site services Argo CD · OpenBao Forgejo mirror read-only Root CA vault room offline HSM · two keyholders Global views Thanos query · Grafana mTLS · SNI Deployment — Three Sites, Twenty Clusters, No Cross-Site Dependency In-Site Application we own Security / platform Interface / broker Data store synchronous A site cut off from the other two still changes, issues and serves for itself. Mirrors, global views and the yearly root ceremony pause. v 1.0 · owner Platform Networking Architecture · date 2026-09 Deployment Architecture Three sites, three rooms per site, 20 clusters, and the site services each site runs for itself. HTML page SVG draw.io
23 Merge Intent change owner repo · 2 approvals Wave 0 Canary cluster synthetic + real Health gate 10 min Wave 1 One cluster per site ≈ 15% of proxies Health gate 15 min Wave 2 Remaining clusters site by site Health gate per site On breach Pin previous bundle one operation · ≤ 60 s breach A Mesh Configuration Change Rolls Out in Waves Data store Application we own Decision point Risk / gap failure / alternate Gates read mesh-attributed error ratio, NACK count and drift. Wave 1 and 2 gates halt and revert the same way; one edge is drawn. v 1.0 · owner Platform Networking Architecture · date 2026-09 Configuration Rollout in Waves A merged mesh change reaches one canary cluster first, then one cluster per site, then the rest, with a health gate between each wave. HTML page SVG draw.io
24 Install Wave 1 · 5% Wave 2 · 30% Wave 3 · 100% Retire Control plane 1.N+1 beside 1.N Both revisions serve Both revisions serve 1.N has no proxies 1.N removed Namespaces Canary tag moved Low-risk namespaces Tier-2 namespaces Tier-1 namespaces Proxies Rolling restart · PDB Rolling restart · PDB Rolling restart · PDB Skew window closed Gate Conformance suite Errors · p99 · NACKs Errors · p99 · NACKs Errors · p99 · NACKs Zero proxies on 1.N Abort Retag to 1.N · restart Retag to 1.N · restart Stop · mixed fleet OK Proxy Fleet Upgrade — Two Revisions Side by Side A newer proxy never talks to an older control plane: each proxy connects to its own revision, so skew is held in both directions by construction. v 1.0 · owner Platform Networking Architecture · date 2026-09 Proxy Fleet Upgrade A new mesh version is installed next to the old one, namespaces move to it in waves, and an abort leaves a mixed fleet that still works. HTML page SVG draw.io
25 Collected Kept Answers Pages on Request metrics Envoy stats · pruned Thanos 15 d · 13 mo Rate, errors, p99 by peer Mesh errors over 0.01% Traces Span per hop · 1% Tempo · 7 d Which hop was slow Context dropped Access logs ALS · all errors ClickHouse · 30 d Peer, route, retries Undeclared destination Config state Sha and nonce per proxy Thanos · current Is intent effective? Stale over 60 s · NACK Identity Expiry · issuance ClickHouse · 13 mo Who got which cert Expiry within 6 h Mesh cost Proxy CPU, memory, bytes OpenCost · monthly What each team pays Over 8% CPU budget Observability — What Is Collected, Kept, Answered and Alerted v 1.0 · owner Platform Networking Architecture · date 2026-09 Observability Six signal types, where each is kept and for how long, the question it answers, and the condition that pages someone. HTML page SVG draw.io
26 Driver Planning figure Lever Reported as Idle proxy floor 40k proxies × idle ≈ 400 vCPU at 10 m Dependency scoping Per team · monthly Request processing 800k proxy hops/s ≈ 80 vCPU Connection reuse vCPU per 1k rps Proxy memory Config size × proxies ≤ 4.8 TB at p95 Scoping · L4 per node p95 · p99 per proxy Telemetry Series × label values Largest line item Sampling per namespace Bytes per team Cross-zone traffic Spillover · failover Measured, not guessed Locality routing Bytes by locality pair Mesh Cost — Where the Budget Goes at 40,000 Proxies At this fleet size the idle floor costs about five times the traffic itself. That is why scoping ships before any latency tuning. v 1.0 · owner Platform Networking Architecture · date 2026-09 Mesh Cost Budget Where CPU, memory and telemetry spend goes at 40,000 proxies, the lever for each, and how it reaches the team that pays. HTML page SVG draw.io
27 MVP · one site SPIRE identity · 24 h Permissive → strict In-proxy authz · dry-run 1% weights · revert Timeouts · retries · ejection Namespace scoping Clusters on one network Phase 2 · three sites Dependency scoping · delta Config waves · auto halt East-west gateways Controlled egress Retry budgets · shedding Revision upgrades Phase 3 · cost and depth Per-node L4 for no-L7 External authz opt-in Cross-site spillover SLO-burn rollback Root rotation rehearsal Policy replay at scale Roadmap — What Each Phase Makes True Security / platform Application we own Interface / broker Opportunity Coarse scoping ships with the MVP. Unscoped, every proxy carries the whole mesh and its memory grows with the fleet instead of with the workload. v 1.0 · owner Platform Networking Architecture · date 2026-09 Roadmap What each phase makes true, in the order that keeps the mesh inside its budget and the fleet reversible. HTML page SVG draw.io
28 Declare subset version label Mirror idempotent only Cohort header match Weight step 1 → 5 → 25 → 50% Read effective mesh-reported share Decide promote or revert Release pods ready no diffs staff pass applied analysis next release Progressive Delivery — The Loop Every Release Runs Application we own Interface / broker Data store Decision point v 1.0 · owner Platform Networking Architecture · date 2026-09 Progressive Delivery Cycle The six steps every release runs through, and the evidence that moves it from one to the next. HTML page SVG draw.io

7 · Assurance

Trust zones, how an authorisation decision is made at the destination, and every named failure with what contains it.
29 Outside the mesh External APIs declared hosts Legacy VM job no identity Boundary Egress gateway TLS origination Strict namespaces checkout default deny ledger allow by identity Mesh control istiod config · no keys SPIRE server cluster CA Key custody OpenBao PKI intermediates Offline root HSM · ceremony TLS · allowlist declared host plaintext refused xDS over mTLS CA renewal Security — Trust Zones and What Crosses Them External / third party Risk / gap Interface / broker Application we own Security / platform synchronous failure / alternate event / async No proxy holds a key that proves any identity but its own. istiod holds no signing key at all; the only in-cluster CA lives 7 days. v 1.0 · owner Security Architecture · date 2026-09 Security — Trust Zones Five zones by exposure, what is allowed to cross between them, and what a compromise in each can reach. HTML page SVG draw.io
30 checkout app checkout proxy ledger proxy ledger app OTel collector 1. POST /refunds 2. mTLS · SVID c07/ns/shop/sa/checkout 3. chain valid · SAN in expected set 4. deny rules first: none match 5. allow rule matches peer, POST, path 6. dry-run rule v2: would deny 7. shadow denial, no header values 8. forward · peer identity header 9. 201 10. 201 11. 201 Authorisation at the Destination — Enforced and Dry-Run Rules Together No step leaves the ledger proxy. The dry-run rule is compiled alongside the enforced one and changes nothing but a log line. v 1.0 · owner Security Architecture · date 2026-09 Authorisation at the Destination One POST from checkout to ledger, showing the order the inbound proxy evaluates identity, deny rules, allow rules and a dry-run rule. HTML page SVG draw.io
31 Detected by Contained by Requests see Back to normal istiod down xDS age per proxy Last-good config held Nothing Push resumes SPIRE or CA down Renewal errors · expiry 24 h SVIDs · 7 d CA Nothing for 12 h Emergency issuance Bad config push Wave gate · NACKs Wave 0 only Errors in one cluster Bundle pinned ≤ 60 s Trust-bundle skew Handshake cause code Additive bundle · 30 d TLS errors, attributed Bundle converges Proxy OOM Container restarts Pod unready · drained Retried on a peer Restart refetches Injection webhook down Admission errors Strict namespaces refuse Nothing · no new pods Webhook back Retry storm retry_overflow Budget 20% · limits Fast 503s Budget refills East-west gateway lost Remote ejections Local-only serving Less remote capacity Gateway returns Clock skew chrony offset over 60 s Node tainted Nothing Resync · untaint Telemetry sink slow Collector drop count Bounded buffers Nothing Gap in telemetry Failure Modes — Detection, Containment, What Requests See v 1.0 · owner Platform Networking Architecture · date 2026-09 Failure Modes The ten failure classes the requirement assumes will happen, how each is noticed, what contains it, and what requests see meanwhile. HTML page SVG draw.io

Architecture One-Pager

The problem, the shape of the answer, the decisions that carry it, and why it should still be right in ten years.

A mesh that is compiled ahead of time and enforced locally: every proxy forwards on three things it already holds, so the control plane can fail without taking a single request with it.

Two thousand services call each other across twenty clusters in three data centres. Today each team decides for itself whether to encrypt, how to authenticate a caller, how long to wait and how often to retry, and the answers are inconsistent in exactly the places where an incident starts. A mesh fixes that by moving those decisions into a proxy beside every workload. It also puts a new component on the path of every request the organisation serves, 40,000 copies of it, and that is the harder problem. A mesh that encrypts everything but goes down with its control plane, or costs 20% of the fleet's CPU, is a regression presented as a platform.

Intent is declared in three Git repositories (routing, authorisation, identity), checked by policy before merge and again at admission, and applied to each cluster by a site-local Argo CD in waves. In each cluster, istiod compiles that intent into per-workload Envoy configuration, scoped to what the workload is declared to call, and streams it as deltas. SPIRE issues each workload a 24-hour X.509 identity derived from what the kubelet reports about it, chained through a 7-day in-cluster CA and a name-constrained per-cluster intermediate in OpenBao to a root that lives on an offline HSM. Envoy sidecars terminate and originate mTLS, evaluate authorisation in-process, route by weight and header, and shed load before it queues. Clusters in different sites meet through east-west gateways that route on SNI and hold no identity. Cilium provides pod networking, BGP-announced gateway addresses and a second, independent enforcement plane. Telemetry leaves every proxy through a bounded node-local buffer into Prometheus and Thanos, Tempo and ClickHouse, tagged with whether a failure came from the application, the mesh or the network.

What it is, and what it is not

Configuration compiled ahead of the requesta control plane, CA or policy service asked anything while a request is being forwarded.
Identity attested by the platforma name the workload chooses, an IP address, or a namespace label anyone with kubectl can set.
Authorisation decided at the destination proxya synchronous call to a central policy server on every request, except where a service opts in and declares the latency.
One writer per kind of intentroutes, policy and weights edited by whoever has cluster-admin, racing each other during an incident.
A proxy per workload, with a node-level L4 option latera shared per-node L7 proxy multiplexing many identities in one process from day one.
A freeze when the control plane is downan outage, and also not a promise that a new deny lands while it is down.

The decisions that are the architecture

01The control plane compiles; it is never consulted

Every proxy holds its last acknowledged configuration, a trust bundle and an unexpired certificate, and forwards on those alone. A total control-plane outage freezes intent; it does not stop traffic.

ADR-01

02Istio on Envoy, with the standard APIs preferred

The most widely deployed open mesh, a CNCF graduated project, with the L7 features the requirement needs today and a per-node L4 mode for later. Routing is written in the Kubernetes Gateway API wherever it has the field.

ADR-02

03SPIRE attests; the workload never names itself

Identity comes from a node attested by projected token and a process the kubelet vouches for. Every issuance is recorded with that evidence for 13 months.

ADR-07

0424-hour certificates are the containment and the outage budget

Renewal at 12 hours gives the fleet 12 hours of issuer outage with no degradation, and a stolen key expires within a day. The two numbers were chosen together.

ADR-08

05A cluster can only mint its own identities

Each cluster intermediate is name-constrained to its trust domain, lives in OpenBao, and is signed once a year at an offline HSM ceremony. Proxies trust roots only.

ADR-09

06Authorisation in the proxy, dry-run first, impact-gated

Policies compile into the destination proxy and cost microseconds. A change is shadowed against live traffic and refused at merge if it would deny more than 0.5% of a service's current calls.

ADR-11

07One owner of weights, and revert is the fastest path

Argo Rollouts is the only writer of canary weights and reads back the share the mesh applied. Reverting a mesh change pins a signed bundle that was already rendered, ahead of Git.

ADR-15

08Every retry is paid for from a budget

Timeouts are mandatory, retries are limited to idempotent methods and 20% of active requests per destination, and deadlines travel with the call so nested retries cannot multiply.

ADR-17

Why it holds up over time

A mesh is adopted for a decade and its components are replaced every two or three years. The design is arranged so that the replacements are local: each seam between components is an open standard or a documented contract, and the decisions that encode organisational judgement (who may change what, what a request depends on, how long a credential lives) are kept out of any one product's configuration language.

The seams are standards, not products

SPIFFE for identity, xDS for proxy configuration, the Kubernetes Gateway API for routes, OTLP for telemetry. Linkerd, Cilium and a future Envoy-based mesh all speak at least three of them, so replacing Istio is a migration of objects rather than a rewrite of the platform's contract with 250 teams.

The dependency direction is the architecture

Nothing on the request path depends on the control plane, the CA or the configuration store. That rule survives a change of proxy, of mesh, even of scheduler. It is what keeps a replacement of any component low-risk, because the component being replaced was never on the path.

The coupled numbers are written down together

Certificate lifetime, renewal point, the outage the fleet survives and the clock-skew budget are one decision, recorded in one place. A future team shortening certificates to an hour will see, in the same record, that they have also turned a 30-minute CA outage into an outage.

Intent outlives the tool that applies it

Routes, policies and dependencies live in Git as reviewed, signed history, with the sha stamped on every applied object and reported back by every proxy. Argo CD, istiod or both can be replaced and the audit trail, the drift join and the separation of duties carry on unchanged.

Upgrades are a routine, not a project

Revision-based upgrades run two mesh versions side by side and move namespaces in waves, every quarter. A platform that upgrades four times a year never falls out of upstream support and never faces a three-version jump that turns into a programme.

The cost model scales with the fleet, not with hope

The idle floor of 40,000 proxies is measured and attributed to teams monthly. When the per-node L4 data plane matures, the saving is a measured decision per namespace, and the architecture already has a place for it.

The root is rehearsed, not remembered

A ten-year root is replaced by people who were not there when it was created. Rotating a staging mesh's root every year keeps the ceremony, the overlap arithmetic and the runbook current long before production needs them.

Failure is assumed, and drilled

Each of the ten failure classes has a detection signal, a containment and a quarterly game-day. Assumptions that are exercised stay true as the system changes; assumptions that are only documented quietly stop being true.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.

QualityTargetHow it is metView
Control plane availability ≥ 99.9% monthly istiod and SPIRE server: three replicas across three rooms per cluster; one sufficient; per-cluster blast radius 22
Mesh-attributable data-path availability ≥ 99.99% monthly; ≤ 0.01% of failed requests caused by the mesh No request-path dependency off the proxy; failure-origin tagging makes the figure measurable rather than argued 11
Survive total control-plane outage ≥ 12 h with no degradation Last-known-good config held indefinitely; 24 h SVIDs renewed at 12 h; SPIRE CA 7 days; Cilium for emergency blocks 21
Added latency per hop p50 ≤ 0.5 ms, p99 ≤ 1.5 ms at ≤ 2,000 rps Scoped config, connection reuse, in-proxy authz, pruned stats; measured per proxy in the proof phase 11
Authorisation decision p99 ≤ 0.2 ms Compiled RBAC filter in Envoy; external authz opt-in only, with a declared budget 30
mTLS handshake, cold p99 ≤ 8 ms, amortised by reuse TLS 1.3, ECDSA P-256 SVIDs, long-lived upstream connection pools 16
Config convergence 95% of affected proxies: p95 ≤ 5 s, p99 ≤ 15 s Delta xDS, dependency scoping, per-cluster istiod; convergence observed by sha, not by sync status 14
Weight change and rollback Weights p95 ≤ 10 s; rollback ≤ 60 s after breach Argo Rollouts analysis every 30 s; one-route delta; revert pins a pre-rendered bundle 18
Certificate issuance p99 ≤ 2 s; 200/s sustained, 2,000/s burst SPIRE server per cluster, agent-side caching, CloudNativePG datastore; upgrade waves rate-limited to issuance headroom 16
Scale 2,000 services, 40,000 proxies, 20 clusters, 400k rps with 3× burst Control plane sharded by cluster (≤ 2,000 proxies per istiod set); config size bounded by declared dependencies 08
Proxy resources Memory p95 ≤ 120 MB, p99 ≤ 200 MB; ≤ 0.1 vCPU per 1,000 rps Namespace scoping in MVP, dependency scoping in Phase 2; concurrency set per workload size 26
Mesh share of fleet ≤ 8% CPU, ≤ 10% memory OpenCost attribution per team, monthly report against budget; per-node L4 for no-L7 namespaces in Phase 3 26
Configuration store durability RPO 0, RTO ≤ 15 min Forgejo on synchronous PostgreSQL standby, read-only mirrors at each site; etcd per cluster holds the applied copy 12
CA material RPO 0; intermediate reissue ≤ 30 min OpenBao Raft across three rooms; root offline on HSM with two keyholders; a spare signed intermediate held sealed per site 09
Retention Metrics 15 d raw / 13 mo rolled up; traces 7 d; access logs 30 d; issuance and audit 13 mo Thanos downsampling on Ceph object storage; Tempo; ClickHouse TTLs; Object Lock bucket for audit 25

Scope

In scope

  • Workload identity, issuance, rotation, revocation and the trust hierarchy from root to SVID
  • mTLS for east-west traffic, permissive and strict modes, and the plaintext report
  • Destination-side authorisation with default deny, dry-run and an impact gate
  • Subset routing, 1% weights, header cohorts, mirroring, locality routing and one-step revert
  • Timeouts, budgeted retries, deadline propagation, outlier ejection and load shedding
  • Sidecar lifecycle, injection policy, configuration scoping, delta distribution and fleet upgrades
  • Multi-cluster service federation through east-west gateways, and controlled egress
  • Per-hop metrics, traces and access logs, failure attribution, drift and cost attribution

Explicitly out of scope

  • The north-south edge: public ingress, WAF, DDoS protection and end-user authentication
  • Workload scheduling, cluster provisioning and node lifecycle
  • Application-level authorisation that needs user entitlements or session state (the external-authz hook is the seam for it)
  • Traffic to databases and brokers that clients reach without passing through a sidecar
  • A self-service portal; changes are reviewed in Git
  • Any managed cloud service; every component runs on hardware the organisation owns

Prove it before production depends on it

Eight weeks, one site, two clusters, 400 synthetic services with a real dependency graph and three real services migrated. Each item below replaces an assumption in this record with a measurement or retires a product behaviour from the list above.

  1. Measure idle CPU and memory per sidecar at 400 and at 2,000 synthetic services, scoped and unscoped
  2. Measure added p50 and p99 per hop at 500, 1,000 and 2,000 rps per proxy, with authz and stats on
  3. Sign 2,000 SVIDs a second from one SPIRE server set on CloudNativePG, and restart 2,000 pods at once
  4. Confirm Envoy refuses an SVID signed outside its intermediate's name constraint
  5. Apply the retry-budget and expected-timeout EnvoyFilter, then upgrade Istio one minor version with it in place
  6. Run a dry-run policy at scale and compare its would-deny log with the impact replay of the same change
  7. Take istiod, SPIRE and OpenBao down for 13 hours under load and record what changes at hours 0, 12 and 13
  8. Fail a canary on purpose and time detection, revert and convergence end to end
  • Pass: mesh-attributed failures stay under 0.01% throughout the 13-hour outage, and the first failures after 12 hours are renewal alerts, not requests.
  • Pass: rollback converges on 95% of caller proxies within 60 seconds of the analysis breach.
  • Fail and rethink: idle CPU per scoped sidecar above 20 millicores, which moves the per-node L4 layer from Phase 3 into Phase 2.

Open risks, carried rather than hidden

RiskIf it landsResponse
A deny cannot land while the control plane is down A revoked authorisation keeps working for as long as istiod is unavailable, up to the 24-hour certificate lifetime Accepted and stated. Emergency blocks use Cilium network policy, which does not depend on istiod; the promise on deny propagation is 15 s p99 with the control plane up (ADR-04)
A crashed proxy cannot restart during a total outage Pods whose sidecar crashes in the outage window stay unready until istiod returns; the requirement asked for a local config cache Deviation recorded in ADR-04. Capacity loss is bounded by crash rate times outage length; forking the proxy agent to persist xDS was rejected as a permanent maintenance cost
Features reached through EnvoyFilter Retry budgets and deadline handling depend on an Envoy-level patch that an Istio upgrade can silently break One generated filter, pinned per revision, covered by the upgrade conformance suite; removed when Istio's API exposes the fields (ADR-17)
Declared dependencies are incomplete A service calling something new is refused by its scoped proxy until the declaration ships, which looks like an outage to the team Undeclared calls fail with a distinct flag attributed to mesh configuration, alert the owning team, and the dependency declaration ships in the same change as the code that needs it (ADR-24)
The idle floor is larger than estimated At 40,000 proxies each extra 10 millicores idle is another 400 vCPU, enough to break the 8% budget on its own Measured first in the proof phase; the per-node L4 data plane for namespaces with no L7 needs is the lever, pulled forward if the measurement demands it (ADR-03)
Upstream changes direction Istio's investment shifts further toward ambient mode, and sidecar mode receives less attention over the life of the platform Routing intent is in the Gateway API, identity in SPIFFE and telemetry in OTLP; moving namespaces to ambient is supported by the same project, and moving meshes is a migration of standard objects (ADR-02, ADR-14)

Architecture Decision Record

Why every component and every technology on these 31 views is what it is, and what each choice costs.

Thirty-two decisions make up this architecture. Everything else on the thirty-one views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the proxy, the mesh project or the team has changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Latencies, resource figures and rates are the requirement's targets or planning estimates, and each is replaced by a measurement in the proof phase. Seven product behaviours are relied on and must be confirmed on the exact releases chosen before production depends on them: Envoy's TLS stack enforcing URI name constraints on a per-cluster intermediate; Istio consuming SPIRE-issued SVIDs over the SPIFFE Workload API with Kubernetes native sidecars; a SPIRE server on CloudNativePG signing 2,000 SVIDs a second; Envoy's retry budget and expected-timeout handling applied through one EnvoyFilter that survives two consecutive Istio minor upgrades; Istio's dry-run mode for authorisation policies, which upstream still marks experimental, behaving as documented at 2,000 proxies per cluster; idle CPU and scoped memory per sidecar at 2,000 services; and OpenBao unsealing from a hardware-backed key on the release deployed.

How to read a record

QuestionThe forcing question: why a decision was needed at all.
ContextThe requirement, the scale and the constraint that make it hard.
DecisionWhat this architecture does, stated so it can be checked.
How it works on-premiseThe concrete mechanism: which package, configured how, on whose hardware.
Options weighedChosen, rejected, deferred, or right elsewhere, with the reason for each.
ConsequencesWhat the choice buys and what it costs, both kept visible.
Choose differently whenThe conditions that would flip the decision for your system.
Why it holds up over timeWhat keeps the decision right as scale, staff and technology change.
LessonThe principle that transfers beyond this platform.

Decision map

Mesh shape 6

What is on the request path, which mesh, where the proxy lives, what a proxy does alone, and how the control plane is divided.

ADR-01The control plane is a configuration compiler, never a request-path dependency ADR-02Istio with Envoy as the mesh, standard APIs preferred over Istio-specific ones ADR-03A sidecar per workload now; a per-node L4 layer later, only where no L7 is needed ADR-04Proxies serve last-known-good configuration indefinitely; certificate lifetime is the only bound ADR-05One control plane per cluster, sharded by cluster, with no cross-site dependency ADR-06Native sidecars for lifecycle ordering; injection failure behaviour declared per namespace

Identity and trust 3

Where a workload's name comes from, how long it lasts, and how one root becomes twenty clusters that cannot vouch for each other.

ADR-07SPIRE issues workload identity from node and workload attestation ADR-0824-hour workload certificates renewed at 12 hours; short lifetime replaces revocation lists ADR-09Offline HSM root, name-constrained per-cluster intermediates in OpenBao, roots-only trust bundle

Transport and authorisation 4

How a fleet moves from plaintext to strict, where the allow-or-deny decision is made, how a policy is proven before it bites, and what leaves the mesh.

ADR-10Move from permissive to strict per service, measured, by reviewed change ADR-11Authorisation is compiled into the destination proxy; external authorisation is a per-service opt-in ADR-12Default deny in strict mode, dry-run before enforcement, and an impact gate at merge ADR-13Egress only to declared hosts through an egress gateway, with Cilium as the second enforcement plane

Traffic and resilience 5

How routes are written, who owns a weight, how fast a mistake is undone, and how retries and overload are kept from becoming outages.

ADR-14Routing intent is written in the Kubernetes Gateway API; Istio APIs only where the standard has no field ADR-15The rollout controller owns traffic weights during a rollout, and judges the share the mesh reports ADR-16Revert pins a previously rendered bundle, and is the one operation allowed ahead of Git ADR-17Timeouts are mandatory, retries are budgeted, and deadlines travel with the call ADR-18Outlier ejection and load shedding at the proxy; locality routing with a declared spillover threshold

Multi-cluster and network 3

How traffic crosses a cluster or a site, how trust between clusters is declared, and what the network under the mesh provides.

ADR-19Flat pod networking inside a site; east-west gateways between sites ADR-20Cross-cluster trust and service export are declared, never default ADR-21Cilium for pod networking, BGP-announced gateway addresses and a second enforcement plane

Change and governance 6

Where intent lives, who may change which part, how a change and an upgrade reach 40,000 proxies, and what is recorded.

ADR-22Intent in three repositories, with separation of duties enforced by repository and by Argo CD project ADR-23Argo CD per site, and every mesh change rolls out in cluster waves behind health gates ADR-24Proxy configuration is scoped to declared dependencies and distributed as deltas ADR-25Admission guards are the second line: OPA Gatekeeper and ValidatingAdmissionPolicy ADR-26Proxy fleet upgrades by control-plane revision, never in place ADR-27An append-only audit trail on Object Lock storage, independent of the SIEM

Observability and cost 5

What every hop reports, how a failure is attributed, how drift is seen, and how the mesh's own cost is kept inside a number.

ADR-28Prometheus per cluster with Thanos for global query and history; labels pruned at the proxy ADR-29Access logs to ClickHouse and traces to Tempo, sampled per namespace, never blocking a request ADR-30Every failure signal carries its origin: application, mesh policy or configuration, or network ADR-31Drift is observed per proxy, not inferred from sync status ADR-32The mesh's cost is a declared budget, measured monthly and attributed to teams with OpenCost

Technology by capability

Every capability on the views, the package or design element that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns; no managed service and no external dependency sits on any request path.

Open source This design
CapabilityChoiceOriginCredible alternativeWhy this oneRecord
Request-path rule Compile ahead, enforce locally This design Central decision per request Control-plane outage becomes a freeze, not an outage ADR-01
Service mesh Istio (sidecar mode) Open source Linkerd; Cilium service mesh Full L7 feature set, multi-cluster, revisions, per-node mode in the same project ADR-02
Data-plane proxy Envoy as a native sidecar Open source Per-node L4 proxy (ztunnel) with waypoints Blast radius of one pod while the platform is young ADR-03
Behaviour with control plane down Last-known-good, bounded by certificate lifetime This design Staleness TTL then fail closed One clock bounds survival, not two ADR-04
Control-plane topology istiod per cluster, multi-primary Open source Primary per site with remote clusters Shards at about 2,000 proxies and contains failure per cluster ADR-05
Proxy lifecycle and injection Kubernetes native sidecars; ValidatingAdmissionPolicy backstop Open source Classic sidecar with startup hold Ordering by the kubelet; strict namespaces never admit unmeshed pods ADR-06
Workload identity issuance SPIRE with k8s_psat and kubelet attestation Open source istiod CA; cert-manager istio-csr Attested identity and issuance evidence, no key in istiod ADR-07
Certificate lifetime 24 h SVIDs, renew at 12 h, no CRL This design 1 h certificates; 30 d with OCSP Gives the 12-hour outage budget and a one-day exposure ADR-08
Clock discipline chrony and node-problem-detector taint at ±60 s Open source PTP everywhere 60 s is ample against a 24-hour certificate ADR-08
Root and intermediates Offline HSM root; name-constrained intermediates in OpenBao Open source Per-cluster roots with federation One root, cryptographic containment per cluster ADR-09
mTLS migration Per-service PeerAuthentication after 7 days of zero plaintext This design Mesh-wide strict on a date Each flip backed by evidence about real callers ADR-10
Authorisation Istio AuthorizationPolicy in Envoy RBAC Open source External authoriser on every call 0.2 ms p99 and no network dependency ADR-11
External authorisation (opt-in) Envoy ext_authz to an OPA sidecar Open source Central policy service Live context only where declared, with a pod-local failure domain ADR-11
Policy proof Dry-run annotation plus access-log replay gate at 0.5% This design Enforce and roll back on complaints Proven against yesterday's and today's traffic before it can deny ADR-12
Egress control ServiceEntry, egress gateway, Cilium egress policy Open source Perimeter firewall only Attributable external calls; bypass-resistant ADR-13
Routing API Kubernetes Gateway API (mesh profile) Open source Istio VirtualService throughout Portable intent for what teams edit most ADR-14
Progressive delivery Argo Rollouts with analysis on Thanos Open source CD pipeline writing weights One writer of weights; judgement on applied share ADR-15
One-step revert Signed OCI bundles in Harbor, pinned by a revert controller This design Git revert and sync Undo does not wait on the pipeline that made the mistake ADR-16
Timeouts, retries, deadlines Mandatory route timeouts, Envoy retry budget 20%, expected-timeout header Open source Attempt counts only Budgets bound the fleet, not just one call ADR-17
Overload and locality Outlier ejection, circuit limits, locality failover at 70% Open source Client-side libraries Shed at the proxy; spillover is declared, not implicit ADR-18
Cross-cluster transport Flat inside a site; SNI east-west gateways between sites Open source Flat network everywhere Shortest path where the network is ours; containment where it is not ADR-19
Federation Exported services and trusted domains declared per cluster This design Merge all discovery by default Trust and discovery are opt-in and reviewable ADR-20
Pod network and load balancing Cilium with BGP control plane and LB IPAM Open source Calico with MetalLB One component for CNI, VIPs and the second enforcement plane ADR-21
Intent store Forgejo, three repositories, signed commits Open source GitLab CE; one monorepo Separation of duties by repository, not by review convention ADR-22
Configuration delivery Argo CD per site, cluster waves with health gates Open source Flux; a central Argo CD No cross-site dependency; waves align with control-plane blast radius ADR-23
Configuration scope Istio Sidecar resources generated from declared dependencies; delta xDS Open source Inferred dependency graph Memory tracks what a workload calls; changes cost what they touch ADR-24
Admission OPA Gatekeeper and ValidatingAdmissionPolicy Open source CI checks only A second line that kubectl cannot skip ADR-25
Mesh upgrades Istio revisions and revision tags, namespace waves Open source In-place upgrade Skew held by construction; abort leaves a working mixed fleet ADR-26
Audit trail Git history, Kubernetes audit, SPIRE audit to Ceph Object Lock Open source SIEM only Append-only for 13 months, independent of the SIEM's retention ADR-27
Metrics Prometheus per cluster with Thanos on Ceph RGW Open source VictoriaMetrics; Grafana Mimir Cluster-local collection, global query, 13-month downsampled history ADR-28
Access logs and traces OpenTelemetry Collector to ClickHouse and Grafana Tempo Open source OpenSearch; Jaeger Columnar logs at 800k hops a second; object-storage traces ADR-29
Failure attribution Failure-origin label from Envoy response flags This design Status codes only Separates the mesh's failures from the application's ADR-30
Drift visibility Drift exporter: acknowledged sha and nonce per proxy This design Argo CD sync status; istioctl on demand Effective state is observed, not inferred ADR-31
Cost attribution OpenCost plus telemetry volume per namespace Open source Spreadsheet estimates The team that meshes a service sees what it costs ADR-32

The decisions, and the alternatives that lost

Mesh shapeWhat is on the request path, which mesh, where the proxy lives, what a proxy does alone, and how the control plane is divided.

ADR-01

The control plane is a configuration compiler, never a request-path dependency

Accepted

What is a proxy allowed to consult while it forwards a request?

Context
A mesh puts a proxy on every request the organisation serves. Whatever that proxy depends on at request time inherits the availability target of every service in the fleet: 99.99% for the mesh-attributable data path. The control plane, the CA and the configuration store are complex, change weekly and target 99.9%. If any of them is consulted per request, even as a cache-miss fallback, the fleet's availability becomes theirs.
Decision
A proxy forwards a request using only three things it already holds: the last configuration it acknowledged, the trust bundle, and an unexpired certificate for its own identity. istiod compiles intent into per-proxy configuration ahead of time and streams it; SPIRE issues certificates ahead of expiry; neither is called while a request is in flight. When they are down, traffic continues on the last good state and new intent waits.
How it works on-premise
Envoy receives listeners, routes, clusters, endpoints and RBAC rules over xDS from istiod and holds them in memory. Certificates and the bundle arrive over SDS from the node's SPIRE agent, which caches them. External authorisation filters are not configured unless a service opts in (ADR-11). A conformance test in the platform pipeline blocks any EnvoyFilter or mesh setting that adds a synchronous call to a control-plane component.
Options weighed
  • ChosenCompile ahead, enforce locally: Control-plane outages become freezes. The mesh's added latency is a property of the proxy alone and can be budgeted in microseconds.
  • RejectedCentral policy or routing decision per request: Can use live state and propagates changes instantly. Puts a network call and its tail latency into every request, and makes its availability the fleet's.
  • RejectedLocal by default, fetch on cache miss: Looks like the best of both. The miss path is exercised only during incidents, which is when the remote service is most likely to be unavailable.
  • Right elsewhereLibrary in every service instead of a proxy: Right for a single-language estate that can redeploy every service for a policy change. Here there are many languages and 250 teams.
Consequences
What it buys
  • A total control-plane outage costs new changes and new workloads, not existing traffic
  • The per-hop latency budget depends only on proxy behaviour, which is measurable in isolation
  • Control-plane components can be upgraded and restarted without a change window for services
What it costs
  • Effective state can diverge from intent for as long as distribution is broken; a revoked permission is not revoked until the push lands
  • Every policy change is a configuration distribution, with its latency and its failure modes
  • Anything that needs per-request live state (a user's entitlements, a risk score) cannot be expressed as mesh policy
Choose differently when
When authorisation must reflect state that changes per request and cannot be compiled, such as per-user entitlements, put that decision in an external authoriser for the services that need it and accept its latency and failure mode explicitly, without moving the rest of the mesh onto that path.
Why it holds up over time
This is a rule about dependency direction, and it does not name a product. A different proxy, a different mesh or a per-node data plane all fit behind it unchanged, and any future design review can test a proposal against one question: does this add something the proxy must ask during a request?
LessonPut what decides on a different availability budget from what serves, and make the serving side able to run on what it already holds.
Shown on views02 07 11 21
ADR-02

Istio with Envoy as the mesh, standard APIs preferred over Istio-specific ones

Accepted

Which open-source mesh carries the requirement for the next decade?

Context
The requirement needs L7 routing with 1% weights, header matching and mirroring, destination-side authorisation on identity with dry-run, outlier ejection, locality failover, cross-cluster federation through gateways that hold no identity, controlled egress and revision-based upgrades. It must be open source, run on-premise, and be supportable by a platform team for years without a vendor subscription.
Decision
Use Istio, a CNCF graduated project, in sidecar mode with Envoy as the proxy and SPIRE as the certificate source (ADR-07). Write routes in the Kubernetes Gateway API where it has the field and use Istio's own APIs only where it does not (ADR-14). Keep Istio's per-node L4 mode (ambient) as the Phase 3 option for workloads with no L7 needs (ADR-03).
How it works on-premise
Istio is installed per cluster with Helm charts pinned by revision, from images mirrored into Harbor and signed with cosign. The platform team builds nothing from source; it tracks upstream minor releases within the support window and applies patch releases within two weeks of publication.
Options weighed
  • ChosenIstio with Envoy sidecars: Broadest L7 feature set, multi-cluster with east-west gateways, revision upgrades, dry-run policy, SPIRE integration, and a per-node mode in the same project.
  • RejectedLinkerd: Smaller, simpler, and its Rust proxy is lighter than Envoy. Since 2024 stable release artifacts are published only by the vendor, and weighted routing, mirroring and egress control are thinner than the requirement needs.
  • RejectedCilium service mesh: Already our CNI, and L4 is excellent. L7 runs in a per-node Envoy shared by many identities, and its mutual authentication does not bind each connection to a workload certificate the way mTLS does.
  • RejectedConsul: Mature and works outside Kubernetes. Relicensed under the Business Source License in 2023, so it fails the open-source constraint.
  • DeferredKuma: Envoy-based and multi-zone by design. Smaller community and CNCF sandbox maturity; worth re-evaluating if Istio's direction changes.
Consequences
What it buys
  • Every capability in the MVP and Phase 2 exists upstream; the platform team configures rather than builds
  • Envoy is the most widely deployed L7 proxy, so its behaviour under failure is well documented by others
  • Ambient mode gives a migration path to lower per-pod cost without changing mesh
What it costs
  • Istio's configuration surface is large and easy to misuse; admission guards and a narrow set of allowed fields are needed (ADR-25)
  • Envoy's memory floor per sidecar is higher than Linkerd's proxy
  • Two features the requirement needs (retry budgets, deadline handling) are reachable only through EnvoyFilter today
Choose differently when
A single-cluster estate that needs mTLS, retries and golden metrics but no weighted routing, mirroring or controlled egress, and is willing to buy vendor support, gets most of the value from Linkerd at a lower resource cost and operational load.
Why it holds up over time
The choice is hedged at every seam: identity is SPIFFE and comes from SPIRE, routes are Gateway API objects, telemetry is OTLP. If Istio loses momentum, the migration is to another mesh that reads the same objects and the same certificates, not a rewrite of what 250 teams have declared.
LessonPick the product for today's requirement, and write your intent in the standards it implements, so that picking again later is a migration and not a rebuild.
Shown on views02 07 08
ADR-03

A sidecar per workload now; a per-node L4 layer later, only where no L7 is needed

Accepted

Should the proxy be one per workload or shared per node?

Context
A sidecar gives per-workload identity, per-workload configuration scope and a blast radius of one pod. It costs a proxy's memory and idle CPU floor multiplied by 40,000 pods, which planning puts at about 400 vCPU idle and up to 4.8 TB of memory at the p95 target. A shared per-node proxy amortises that cost but multiplexes many identities in one process and fails a whole node at a time. A split, with L4 identity and encryption per node plus L7 per service only where needed, buys most of the saving for a second data-plane component.
Decision
Run Envoy sidecars for every meshed workload in the MVP and Phase 2. Every MVP capability, from subset routing to destination authorisation on path, is L7. In Phase 3, offer Istio's ambient mode (a per-node L4 proxy, ztunnel, with per-namespace L7 waypoint proxies) to namespaces that declare no L7 policy, once the proof phase has measured the saving and the node-level failure mode has been drilled.
How it works on-premise
Sidecars are injected as Kubernetes native sidecars. Proxy concurrency is set from the pod's CPU request, not left at the node's core count. The Phase 3 pilot moves batch and internal-tooling namespaces first, where a node-wide proxy failure is cheapest; the attribution in ADR-32 compares their cost per request before and after.
Options weighed
  • ChosenSidecar everywhere, per-node L4 opt-in later: Smallest blast radius and simplest identity model while the platform is new; the cheaper model arrives when it can be measured.
  • DeferredPer-node L4 plus waypoints from day one: Lower cost at scale. Adds a second data-plane component and a node-wide failure mode before the team has operated either.
  • RejectedShared per-node L7 proxy: Best resource amortisation. One process holds many workloads' keys, which contradicts the requirement that a compromised proxy can impersonate only its own workload.
  • RejectedSidecar only for services that opt in: Cheap, and leaves mTLS and identity optional exactly where teams have not thought about them.
Consequences
What it buys
  • A proxy crash or compromise affects one pod and one identity
  • Configuration can be scoped per workload, which keeps each proxy's memory tied to what it calls
  • One data-plane model to operate, debug and upgrade in the first two phases
What it costs
  • Idle floor multiplied by pod count is the largest single line in the mesh budget
  • Pod startup includes proxy readiness, adding roughly a second to every pod start
  • Two data-plane models coexist from Phase 3, with different debugging tools
Choose differently when
When pod density is high (dozens of small pods per node), most traffic needs only mTLS and L4 policy, and the organisation has run node-level agents like a CNI at scale without incident, start with the per-node model and add L7 proxies per service.
Why it holds up over time
The decision is expressed per namespace, not per fleet. As per-node data planes mature, namespaces move one at a time on measured cost, and nothing else in the architecture changes: identities are still SPIFFE, policies are still authored the same way, and telemetry keeps the same labels.
LessonBuy the simplest failure model first and the cheapest resource model second, and make the switch a per-tenant setting rather than a platform migration.
Shown on views08 26 27
ADR-04

Proxies serve last-known-good configuration indefinitely; certificate lifetime is the only bound

Accepted

How should a proxy behave when the control plane is gone?

Context
The options are to serve last-known-good forever, to serve it until a staleness timer then fail closed, or to keep existing connections but refuse new ones. Serving forever maximises availability and lets the fleet diverge silently from intent, including a revoked permission that never takes effect. A timer bounds divergence but turns a long control-plane outage into a data-path outage. Certificate lifetime already bounds how long a proxy can run alone.
Decision
Proxies keep their last acknowledged configuration with no staleness timer. The 24-hour certificate is the bound: after 12 hours without issuance, renewals fail and pages fire; after 24, pods fail closed one by one. The mesh promises that a deny propagates within 15 s p99 while the control plane is up and makes no promise while it is down. Emergency blocks during an outage use Cilium network policy, which does not depend on istiod.
How it works on-premise
No xDS staleness setting is configured on Envoy. The drift exporter reports xDS age per proxy, and a page fires when istiod has been unreachable for 5 minutes in any cluster. A crashed Envoy restarts under its container's restart policy and waits for istiod, while the pod stays unready and is removed from endpoints. The emergency-block runbook applies a CiliumNetworkPolicy selecting the pod by label, through the Kubernetes API, which is a separate failure domain from istiod.
Options weighed
  • ChosenServe indefinitely, bounded by certificate lifetime: No second timer that could expire first; availability during an outage is as long as identity allows.
  • RejectedStaleness TTL, then fail closed: Bounds divergence. A TTL shorter than the certificate converts a control-plane outage into a fleet outage; a longer one does nothing.
  • RejectedPersist xDS to local disk so crashed proxies restart without istiod: Meets the requirement's proxy-crash row literally. Requires forking or extending the proxy agent and carrying that patch through every upgrade.
  • RejectedRefuse new connections when stale: Protects against stale policy on new callers, and breaks every connection pool that recycles during the outage.
Consequences
What it buys
  • A 12-hour total control-plane outage causes no request failures
  • One number, the certificate lifetime, governs how long the fleet can run alone
  • The deny-propagation promise is explicit, so security incident runbooks do not assume a push lands during an outage
What it costs
  • A permission revoked during an outage keeps working until the control plane returns or the certificate expires
  • Pods whose sidecar crashes during a total outage stay out of service until istiod returns; this is a recorded deviation from the requirement
  • An emergency block during an outage is coarse (L3/L4 by pod label) rather than a precise L7 deny
Choose differently when
Where a revoked permission taking effect matters more than availability (a payment authorisation boundary under regulatory control, for example), put that boundary behind an external authoriser with fail-closed posture, or shorten certificates for those workloads, and accept the outage that follows.
Why it holds up over time
The rule does not depend on how configuration is delivered or how long certificates last; it depends only on there being one bound instead of two. If certificate lifetimes change, the survivable outage changes with them in the same record, and nothing else needs retuning.
LessonDo not add a timer that fails before the one you already have. Decide which clock bounds survival, and state what the system will not promise while that clock runs.
Shown on views21 31 06
ADR-05

One control plane per cluster, sharded by cluster, with no cross-site dependency

Accepted

How is the control plane divided so that no replica holds the whole fleet and no region depends on another?

Context
At the design ceiling, 40,000 proxies hold long-lived xDS streams. A single mesh-wide control plane would make one deployment the blast radius for every cluster and put cross-site latency into every configuration push. The requirement asks for multiple replicas per cluster, horizontal distribution, and no cross-region dependency for in-region configuration.
Decision
Each of the 20 clusters runs its own istiod (three replicas across three rooms) and its own SPIRE server set. Clusters are peers in an Istio multi-primary topology: each reads its own API server for local intent and endpoints, and reads peer clusters only for services those peers export (ADR-20). A cluster's control plane serves at most that cluster's proxies, about 2,000.
How it works on-premise
istiod runs on the cluster's platform node pool with a PodDisruptionBudget of one and a HorizontalPodAutoscaler on xDS connection count. Proxies connect through a cluster-local Service, so load spreads across replicas and a replica loss moves only its connections. Remote-cluster access uses a read-only Kubernetes service account per peer.
Options weighed
  • ChosenPer-cluster control plane, multi-primary: Blast radius of one cluster, natural sharding at about 2,000 proxies, and no site depends on another for its own configuration.
  • RejectedOne primary control plane per site, remote clusters: Fewer istiod deployments. A site's primary becomes the blast radius for up to seven clusters and a single upgrade event for all of them.
  • RejectedOne global control plane: Simplest to reason about. Cross-site latency on every push and a fleet-wide blast radius.
Consequences
What it buys
  • A bad istiod release or a runaway push is contained to one cluster, which is also the unit of rollout waves
  • Each site keeps changing and serving during a site partition
  • Control-plane capacity grows linearly with clusters, never with the fleet as a whole
What it costs
  • Twenty control planes to upgrade, monitor and keep on compatible versions
  • Remote-cluster discovery means each istiod watches peer API servers, which is load and credentials to manage
  • Consistency across clusters is eventual; a change lands cluster by cluster by design
Choose differently when
With three or four clusters in one site on one network, a single primary control plane with remote clusters is simpler to run, and the blast-radius argument is weak because the clusters already share failure domains.
Why it holds up over time
Sharding by the unit that already owns failure (the cluster) means growth adds clusters rather than making any one component bigger. The design ceiling can be passed by adding a site without revisiting the control-plane topology.
LessonShard the control plane along the boundary that already contains failures, so one partitioning serves scale, blast radius and rollout at once.
Shown on views08 22 23
ADR-06

Native sidecars for lifecycle ordering; injection failure behaviour declared per namespace

Accepted

How does a proxy start before its application and stop after it, and what happens when injection is unavailable?

Context
Classic sidecars race the application: an app that sends a request before Envoy is ready fails, and a proxy that exits before in-flight requests finish drops them. Injection is done by a mutating webhook served by istiod, so istiod being down means new pods either start unmeshed or do not start at all. The requirement forbids admitting a pod silently unmeshed into a strict namespace, and forbids a global default for the behaviour.
Decision
Inject Envoy as a Kubernetes native sidecar (an init container with restartPolicy Always), so the kubelet starts it and waits for its readiness before starting the application, and terminates it only after the application containers exit. Declare injection failure per namespace: strict namespaces fail closed, permissive namespaces fail open. An in-apiserver admission policy, independent of the webhook, refuses any pod without a sidecar in a strict namespace unless it carries a reviewed opt-out.
How it works on-premise
Two webhook configurations are registered with namespace selectors: one with failurePolicy Fail for namespaces labelled mesh.corp/mode=strict, one with Ignore for permissive ones. A ValidatingAdmissionPolicy (CEL, evaluated inside the API server, so it has no webhook to fail) checks for the istio-proxy init container in strict namespaces. Opt-outs are an annotation that admission accepts only when the pod's namespace lists that workload in the security repository.
Options weighed
  • ChosenNative sidecars plus per-namespace failure policy and an independent admission check: Lifecycle ordering done by the kubelet; injection outages degrade differently where the risk differs.
  • RejectedClassic sidecar with holdApplicationUntilProxyStarts and preStop sleeps: Works on older Kubernetes. Relies on timing hacks that fail under slow nodes and long drains.
  • RejectedOne global webhook failure policy: Simple. Either every namespace stops scheduling when istiod is down, or strict namespaces admit unmeshed pods.
  • DeferredCNI-based injection with no webhook: Removes the webhook dependency. Ambient mode does this for L4; it arrives with ADR-03's Phase 3 pilot.
Consequences
What it buys
  • No startup or shutdown race between application and proxy
  • A strict namespace cannot receive an unmeshed pod even if the webhook is misconfigured
  • Permissive namespaces keep scheduling during an istiod outage
What it costs
  • Requires Kubernetes with native sidecars (stable since 1.33) on every cluster
  • Strict namespaces cannot scale up during an istiod outage in that cluster
  • Opt-outs are a list the security team must keep short and current
Choose differently when
On a Kubernetes version without native sidecars, use the classic ordering settings and a longer termination grace period, and plan the cluster upgrade as a prerequisite for strict mode rather than as a nice-to-have.
Why it holds up over time
Lifecycle ordering has moved from mesh-specific workarounds into Kubernetes itself, which is where it will stay. The failure-policy split is expressed as namespace labels and an admission policy in core Kubernetes, and would apply unchanged to any injector.
LessonLet the platform that owns process lifecycle own proxy ordering, and put the safety check for your most important invariant somewhere that cannot fail with the thing it checks.
Shown on views08 31 04

Identity and trustWhere a workload's name comes from, how long it lasts, and how one root becomes twenty clusters that cannot vouch for each other.

ADR-07

SPIRE issues workload identity from node and workload attestation

Accepted

Where does a workload's cryptographic name come from?

Context
The requirement insists that identity be derived from an attestation of the platform, never from a value the workload asserts, and that every issuance be recorded with the attestation that justified it. Istio's built-in CA authenticates a certificate request with the pod's projected service-account token, which is platform-issued but does not bind the request to an attested node, and its issuance logging is not designed as audit evidence.
Decision
Run a SPIRE server set per cluster and a SPIRE agent on every node. The agent attests its node with a projected service-account token checked against the API server; the agent then attests each workload by asking the kubelet which pod owns the requesting process. SVIDs are issued for identities of the form spiffe://<cluster>.mesh.corp/ns/<namespace>/sa/<service-account>. Envoy obtains its SVID and trust bundle over SDS from the SPIFFE Workload API socket. Registration entries are generated from Kubernetes objects, not written by hand.
How it works on-premise
SPIRE server uses the k8s_psat node attestor, a CloudNativePG PostgreSQL datastore with a synchronous standby, and the UpstreamAuthority plugin for Vault-compatible PKI pointed at its cluster's OpenBao mount. spire-controller-manager creates entries from a ClusterSPIFFEID template. The SPIFFE CSI driver mounts the Workload API socket into each pod. Server audit logging is enabled and shipped to ClickHouse as issuance records, with a nightly reconciliation against the datastore.
Options weighed
  • ChosenSPIRE with node and workload attestation: Identity is bound to an attested node and a kubelet-verified process; issuance evidence is first-class; SPIFFE is the standard every mesh reads.
  • Rejectedistiod's built-in CA with a plugged-in intermediate: Simplest, and fast. Puts a signing key inside the configuration compiler and gives weaker attestation evidence.
  • Rejectedcert-manager with istio-csr and an OpenBao issuer: Keeps keys out of istiod and is familiar to Kubernetes teams. Attestation is still token-only, and issuance at 2,000 a second is a load on the Kubernetes API through CertificateRequest objects.
Consequences
What it buys
  • A pod cannot obtain a certificate for an identity other than the one the kubelet reports for it
  • istiod holds no signing key; compromising configuration does not mint identities
  • The same identities work for non-Kubernetes workloads later, through SPIRE's other attestors
What it costs
  • A second control-plane component per cluster, with its own datastore to back up
  • Istio's SPIRE integration is less common than its built-in CA, so fewer people have debugged it
  • Issuance throughput at mass restart depends on the datastore and must be proven (evidence note)
Choose differently when
In a single cluster with no audit requirement for issuance evidence and no plan to extend identity beyond Kubernetes, Istio's built-in CA with an intermediate from an offline root gives most of the security for less to operate.
Why it holds up over time
SPIFFE identities and the Workload API are standards that Envoy, Istio, Linkerd, Cilium and gRPC all consume. Whatever the mesh becomes, the identity plane can stay, and the thing an auditor asks for (who got which certificate, on what evidence) does not depend on the mesh at all.
LessonKeep the component that issues identity separate from the component that distributes configuration, and make issuance evidence something the issuer writes rather than something you reconstruct.
Shown on views09 16 12
ADR-08

24-hour workload certificates renewed at 12 hours; short lifetime replaces revocation lists

Accepted

How long should a workload certificate live?

Context
Short lifetimes shrink the value of a stolen key and make revocation lists unnecessary; they also multiply issuance load, tighten the clock-skew budget and shorten the control-plane outage the fleet survives. A 1-hour certificate means a 30-minute CA outage is an outage. Long lifetimes invert that and require a working revocation mechanism, which in practice means OCSP or CRL checks on the request path.
Decision
Workload SVIDs live 24 hours and are renewed at 12 hours, the requirement's stated assumption, adopted because it produces the 12-hour outage survival the availability target needs. There are no revocation lists. Containment of a compromised identity is: an emergency deny on the principal, banning the node's SPIRE agent so renewal stops, deleting the pod, and waiting at most 24 hours for the certificate to expire. Clock skew is held to ±60 seconds and a node beyond it is tainted.
How it works on-premise
SPIRE's default X509-SVID TTL is set to 24 h; agents renew at half-life. chrony on every node syncs to three site-local stratum-1 servers. node-problem-detector reads chrony's offset and taints a node NoSchedule and NoExecute beyond 60 s. Issuance at 40,000 SVIDs a day is about 0.5 a second on average; the design point is the 2,000-a-second restart burst.
Options weighed
  • Chosen24 h lifetime, renew at 12 h, no revocation lists: 12 hours of issuer outage with no effect, a day's exposure for a stolen key, modest issuance load.
  • Rejected1 h lifetime, renew at 30 min: A stolen key is useless in an hour. A 30-minute CA outage becomes a fleet outage, and issuance load is 24 times higher.
  • Rejected30-day lifetime with OCSP or CRLs: Negligible issuance load. Revocation checking on the request path is exactly the dependency ADR-01 forbids, and without it a stolen key lives a month.
Consequences
What it buys
  • The fleet survives 12 hours of total issuer outage with no request failures
  • No revocation infrastructure to operate, and nothing on the request path to check it
  • Clock skew is a non-issue at ±60 s against a 24-hour window
What it costs
  • A stolen key is valid for up to 24 hours; containment relies on authorisation deny and pod deletion in that window
  • A CA outage longer than 24 hours is a fleet outage, so the emergency issuance path must be drilled
  • Lifetime and survivable outage cannot be changed independently
Choose differently when
Workloads handling highly sensitive keys (a signing service, a payment HSM proxy) should run shorter lifetimes, such as 4 hours renewed at 2, in their own SPIRE entry, and accept a shorter survivable outage for those workloads only.
Why it holds up over time
The record names the coupling explicitly: lifetime, renewal point, outage budget and skew budget are one decision. A future change to any one of them has to be made here, where the other three are visible, which is the only reliable defence against a well-meant change that quietly shortens the fleet's survival.
LessonTreat credential lifetime as an availability decision as well as a security one, and write both consequences in the same place.
Shown on views09 16 21 06
ADR-09

Offline HSM root, name-constrained per-cluster intermediates in OpenBao, roots-only trust bundle

Accepted

How does one mesh-wide root become twenty clusters, none of which can mint identities for another?

Context
A single mesh-wide root makes cross-cluster verification simple. If every cluster's issuer can sign any identity, compromising one cluster compromises the mesh. The requirement asks for a mesh-wide root, a per-cluster intermediate with independent lifetime and revocation, the root key offline or in an HSM with a rotation ceremony, reissue of an intermediate within 30 minutes, and a root rotation that is additive with a 30-day overlap.
Decision
The root key is generated in and never leaves an HSM kept offline in a vault room, used once a year with two keyholders present to sign intermediates. Each cluster has a one-year intermediate with an X.509 name constraint that permits only its own trust domain, held in a dedicated PKI mount in its site's OpenBao. SPIRE in that cluster obtains a 7-day signing CA from that mount. Proxies trust roots only. Each site holds one pre-signed spare intermediate per cluster, sealed, so an intermediate can be replaced in under 30 minutes without a root ceremony.
How it works on-premise
The ceremony runs on an air-gapped workstation using OpenSSL with a PKCS#11 provider against the HSM, from a scripted runbook; outputs are signed intermediates and an updated bundle carried on write-once media. OpenBao runs five Raft voters across three rooms per site, auto-unsealed from a hardware-backed key, with one PKI mount and one SPIRE role per cluster. Root rotation publishes the new root into the SPIRE bundle 30 days before any intermediate chains to it (view 17).
Options weighed
  • ChosenOffline root, name-constrained per-cluster intermediates, roots-only bundle: Cryptographic containment per cluster, simple verification everywhere, and intermediates that rotate without touching bundles.
  • RejectedSeparate root per cluster with explicit bundle federation: Strongest isolation. Twenty roots to rotate and a bundle matrix to distribute for every cross-cluster call.
  • RejectedOnline root in OpenBao: Operationally easy. A network-reachable root is the one key whose compromise cannot be contained.
  • RejectedTrust bundle containing intermediates: Lets a proxy verify shorter chains. Every intermediate rotation then becomes a fleet-wide bundle change.
Consequences
What it buys
  • A compromised cluster can mint only its own identities, even though all chains end at one root
  • OpenBao can be unavailable for three days before any SPIRE CA expires
  • Intermediate replacement is a local operation with a sealed spare, not an emergency ceremony
What it costs
  • Relies on Envoy enforcing URI name constraints, which must be proven on the chosen release
  • The yearly ceremony needs two keyholders physically present and a rehearsed runbook
  • A sealed spare per cluster per site is key material that must be inventoried and audited
Choose differently when
If name-constraint enforcement cannot be proven, keep one root but enforce trust domain in every authorisation policy (principals include the cluster), and plan a move to per-cluster roots with SPIFFE federation for clusters with different risk profiles.
Why it holds up over time
Hierarchy depth and lifetimes are arranged so that the rarest operation (root rotation, once a decade) is the only one that touches every proxy, and it is additive. Everything that changes more often happens below the bundle. That shape survives changes of HSM vendor, PKI software and mesh.
LessonArrange a certificate hierarchy so that the operations you do often never touch the thing every client trusts, and the one you do rarely is rehearsed.
Shown on views09 17 22 29

Transport and authorisationHow a fleet moves from plaintext to strict, where the allow-or-deny decision is made, how a policy is proven before it bites, and what leaves the mesh.

ADR-10

Move from permissive to strict per service, measured, by reviewed change

Accepted

How does a fleet of 2,000 services stop accepting plaintext without an outage?

Context
Turning on strict mTLS everywhere at once breaks every caller that is not yet meshed: batch jobs, legacy VMs, health probes from outside the mesh. Leaving everything permissive forever means the encryption promise is not kept. The requirement asks for permissive mode per service, strict mode per service, namespace or mesh, a per-service plaintext report, and a transition that is a reviewable change rather than a switch.
Decision
Every namespace starts permissive when meshed. A service moves to strict through a PeerAuthentication change in the security repository, reviewed by its owners, after the plaintext report has shown zero plaintext for seven days. Namespace-wide strict is allowed only when every service in it is already strict. Mesh-wide strict is not a goal date; it is the state reached when the report shows no permissive services left.
How it works on-premise
Istio's standard metrics label every inbound request with its connection security policy (mutual TLS or none). A Thanos recording rule computes plaintext share per destination service daily, and the report lists plaintext sources as workload identities where the source is meshed and as 'no identity' with namespace and pod otherwise. Health probes are rewritten by the injector to go through the sidecar so they do not appear as plaintext.
Options weighed
  • ChosenPer-service strict after a measured zero, by reviewed change: Each flip is backed by data about that service's actual callers and can be reverted in one operation.
  • RejectedMesh-wide strict on a date: Fast and simple to communicate. Every unknown caller becomes an incident on the same day.
  • RejectedPermissive indefinitely, encryption reported only: No breakage. Plaintext stays accepted wherever nobody chooses to act.
Consequences
What it buys
  • Each strict flip is justified by a week of evidence for that service
  • Unknown callers are found by the report, not by the outage
  • Progress toward full strict mode is measurable and visible per team
What it costs
  • Full strict mode takes months, and permissive services accept plaintext until then
  • The plaintext report depends on telemetry, which is loss-tolerant; zero means zero observed
  • Teams must act; a platform team cannot finish the migration alone
Choose differently when
A greenfield estate with no unmeshed callers should start strict in every namespace and skip permissive entirely; the migration machinery exists for estates that were not born inside a mesh.
Why it holds up over time
The mechanism, observe then enforce per unit, applies unchanged to every future tightening: TLS versions, cipher suites, or post-quantum key exchange when the proxies support it. The report and the reviewed flip become the platform's general tool for raising a floor without an outage.
LessonRaise a security floor one tenant at a time on evidence about that tenant, and make the evidence the gate rather than the calendar.
Shown on views04 29 25
ADR-11

Authorisation is compiled into the destination proxy; external authorisation is a per-service opt-in

Accepted

Where is the allow-or-deny decision evaluated?

Context
In-proxy evaluation costs microseconds and cannot consult anything the configuration does not already contain; a policy change is a configuration push with that latency. An external authoriser can use live state such as a session or a risk score, and puts a network call, its latency and its availability into every request, forcing a fail-open or fail-closed choice with no good answer. The requirement sets 0.2 ms p99 for the in-proxy decision and asks for external hooks only as opt-in with a declared budget and posture.
Decision
Authorisation policies are Istio AuthorizationPolicy objects compiled into Envoy's RBAC filter at the destination proxy, matching on the authenticated peer identity, namespace, method, path and declared request attributes, never on IP address. In Phase 3, a service may add Envoy's external authorisation filter pointing at an OPA sidecar or a service, declaring in the security repository its timeout, whether it fails open or closed, and which paths it covers.
How it works on-premise
Policies are evaluated in Istio's order: custom (external), then deny, then allow. The external option runs OPA with the Envoy plugin as a sidecar, so the network call is to localhost and its failure domain is the pod; policy bundles for OPA come from Harbor. A service's declaration is validated at admission: no external authz filter without a declared timeout under 5 ms and an explicit failure posture.
Options weighed
  • ChosenIn-proxy RBAC, external authz opt-in per service: Microsecond decisions with no dependency for the common case; live-state decisions only where a service has chosen to pay for them.
  • RejectedExternal authoriser for every request: One policy engine with full context. Puts a network call and its failure mode on every request in the fleet.
  • RejectedAuthorisation in the application only: Full context and no mesh dependency. The requirement is that caller identity be a verifiable input to a decision teams do not have to implement.
  • Right elsewhereNetwork policy only (L3/L4): Right as a second layer (ADR-13). Cannot see method or path, and pod identity by label is weaker than by certificate.
Consequences
What it buys
  • Authorisation adds no network dependency and stays within 0.2 ms p99
  • Policies survive rescheduling and IP reuse because they name identities
  • Services that need live context can have it without imposing it on the fleet
What it costs
  • A policy change takes a configuration push to land, typically seconds, never while istiod is down
  • Mesh policy cannot express per-user or per-session rules
  • Services that opt into external authz own the choice between fail-open and fail-closed
Choose differently when
If most services need decisions on per-user entitlements (a multi-tenant SaaS where every call is tenant-scoped), invest in a local external authoriser with pushed data, such as OPA sidecars fed by bundles, as the default, and accept a pod-local dependency on every request.
Why it holds up over time
Identity-based, destination-side authorisation is the model the Gateway API policy work and every major mesh share. Policies expressed on SPIFFE identities and request attributes carry across proxies, and the opt-in seam for live-state decisions is Envoy's external authorisation protocol, which is widely implemented.
LessonMake the common decision local and free, and make the expensive one explicit, per service, with its failure mode written next to it.
Shown on views11 30 29
ADR-12

Default deny in strict mode, dry-run before enforcement, and an impact gate at merge

Accepted

How is an authorisation policy proven correct before it can deny real traffic?

Context
A policy that denies a legitimate caller is an outage that looks like a security success. Default deny makes that risk larger, because forgetting an allow is now fatal. The requirement asks for default deny in strict mode with explicit allows, a dry-run posture reporting what would have been denied, and refusal of any change that would deny more than a declared share of current traffic without an override.
Decision
When a service goes strict, its namespace receives an empty-rule allow policy that denies everything not explicitly allowed; the service's allow policy must merge in the same change. Any new or changed policy is first merged in dry-run mode and runs against live traffic for at least 24 hours. At merge, CI replays the last 24 hours of that service's access logs against the proposed policy; if it would deny more than 0.5% of requests, the merge needs an override approved by a second security owner.
How it works on-premise
Dry-run uses Istio's istio.io/dry-run annotation, which compiles the policy into Envoy as shadow rules and records would-be denials in access logs and metrics without denying. The impact replay is a small Go tool in the security repository's pipeline that queries ClickHouse for the service's inbound requests (peer identity, method, path) and evaluates them against the proposed rules with the same semantics. Overrides are recorded as a signed commit trailer.
Options weighed
  • ChosenDefault deny, dry-run, replay gate: Two independent proofs, one against history and one against live traffic, before a policy can deny anything.
  • RejectedDefault allow with deny lists: No outages from forgotten allows. Anything nobody thought to deny is permitted, which is the opposite of the requirement.
  • RejectedEnforce immediately, roll back on complaints: Fastest. Makes callers discover policy errors through failed requests.
Consequences
What it buys
  • A policy that would break a real caller is caught before merge or during dry-run
  • The would-deny list names callers by identity, which turns a policy review into a caller review
  • The 0.5% threshold makes large-impact changes a deliberate two-person decision
What it costs
  • A day of dry-run delays every enforcement change except emergency denies
  • The replay depends on 24 hours of access logs, which are sampled for successes (ADR-29); rare callers can be missed
  • Istio marks dry-run as experimental; its behaviour must be re-verified on each upgrade
Choose differently when
For an emergency deny against a compromised identity, skip dry-run and the gate entirely: the change type is labelled emergency, applied to all clusters at once and reviewed after the fact.
Why it holds up over time
Shadow evaluation and replay against recorded traffic are techniques, not features of one mesh. If the dry-run annotation disappears, the replay gate still stands, and the platform's promise that no policy bites before it is proven does not change.
LessonTest a rule that can deny traffic against the traffic it will see, both what happened yesterday and what happens today, before it is allowed to deny.
Shown on views04 14 30
ADR-13

Egress only to declared hosts through an egress gateway, with Cilium as the second enforcement plane

Accepted

How does traffic leave the mesh, and what stops a workload going around its proxy?

Context
A sidecar can be bypassed by a process that has the privileges to change its pod's network rules, and an undeclared external destination is a common sign of misconfiguration or compromise. The requirement asks for egress to declared destinations through a controlled path, and for reporting, optionally refusing, traffic to undeclared ones.
Decision
External destinations are declared as ServiceEntry objects in the routing repository and reached through a per-cluster egress gateway that originates TLS. Strict namespaces run with outbound traffic restricted to the registry; permissive ones allow any destination and report undeclared ones. Cilium network policy independently allows egress from workload namespaces only to in-cluster destinations and the egress gateway, so bypassing the sidecar does not reach the outside.
How it works on-premise
Undeclared destinations appear in Envoy metrics under the passthrough and blackhole clusters and in access logs with the original destination; an alert names the source identity. Pod Security Admission at the restricted level denies NET_ADMIN and privileged containers in workload namespaces. The egress gateway runs on the gateway node pool with its own identity, and its outbound source addresses are the only ones perimeter firewalls allow.
Options weighed
  • ChosenDeclared hosts via egress gateway, Cilium as backstop: One path out, visible and auditable, with a second plane that does not depend on the sidecar being intact.
  • RejectedSidecar-only egress control: Simpler. Any process that can alter the pod's traffic redirection escapes it.
  • RejectedPerimeter firewall only: Already exists. It sees node addresses, not workload identities, so it cannot say which service called out.
Consequences
What it buys
  • Every external call is attributable to a workload identity
  • A bypassed or compromised sidecar still cannot reach undeclared external hosts
  • Perimeter firewall rules shrink to the egress gateways' addresses
What it costs
  • Every new external dependency needs a declaration merged before it works in strict namespaces
  • The egress gateway is a shared component whose capacity must follow external call volume
  • Two policy systems (Istio and Cilium) must agree about what is allowed out
Choose differently when
Where workloads legitimately call thousands of changing external hosts (a web crawler, a webhook delivery service), put them in a dedicated namespace with their own egress path and domain-level filtering rather than individual ServiceEntries.
Why it holds up over time
Two independent enforcement planes (identity-aware L7 and label-aware L3/L4) protect against the failure of either, including a future where one of them is replaced. The declarations are plain Kubernetes objects in Git, so the list of what the organisation talks to outlives the tools that enforce it.
LessonControl the way out in two places that fail independently, and keep the list of allowed destinations as reviewed data.
Shown on views07 29 25

Traffic and resilienceHow routes are written, who owns a weight, how fast a mistake is undone, and how retries and overload are kept from becoming outages.

ADR-14

Routing intent is written in the Kubernetes Gateway API; Istio APIs only where the standard has no field

Accepted

In what language do 250 teams declare their routes?

Context
Routes are the mesh configuration teams write most often and change most. Istio's VirtualService and DestinationRule are expressive and specific to Istio. The Kubernetes Gateway API, through its mesh (GAMMA) profile, is a multi-implementation standard for service-to-service routing that Istio, Linkerd, Cilium and others implement, with weighted backends, header matches and request mirroring. Istio's per-node L7 proxies (waypoints) are configured only through it.
Decision
Routes, weights, header cohorts, mirroring and timeouts are written as Gateway API HTTPRoute (and GRPCRoute) objects attached to a Service. Subsets are separate Kubernetes Services selecting a version label, referenced as weighted backends. Connection pool, outlier detection, locality and TLS settings, which the standard does not yet cover, are written as Istio DestinationRules. Every Istio-specific object is listed in an allowlist in the routing repository with the Gateway API issue that would replace it.
How it works on-premise
Gateway API CRDs are installed per cluster at a pinned standard-channel version matching the Istio release. A generator in the routing repository creates per-version Services from a short service manifest, so teams write one file per service. Admission guards reject VirtualService objects outside the allowlist, so the two APIs cannot both route the same service.
Options weighed
  • ChosenGateway API first, Istio APIs for gaps: Portable intent for the objects teams touch most, full capability for the rest, and the path Istio's own per-node mode requires.
  • RejectedIstio VirtualService and DestinationRule throughout: Most expressive today and most documented. Ties every team's routes to one mesh's API.
  • RejectedA platform-defined abstraction CRD that generates both: Smallest interface for teams. Creates an in-house API with no community, documentation or tooling, owned by us forever.
Consequences
What it buys
  • The routes teams write are portable across mesh implementations
  • Moving a namespace to per-node L7 proxies later needs no route rewrite
  • Gateway API's role separation (route owners versus infrastructure owners) matches ADR-22
What it costs
  • Two APIs to learn during the transition, and an allowlist to maintain
  • Some fields (retry details, mirroring percentages) sit in the experimental channel on current releases
  • Per-version Services add objects per service, generated rather than hand-written
Choose differently when
If the estate will never run another mesh and needs features only VirtualService offers today (complex fault injection, rich retry predicates) across most services, VirtualService throughout is simpler for the next three years.
Why it holds up over time
Standards absorb the features that most implementations agree on, so the allowlist of Istio-specific objects should shrink with each Gateway API release. Every entry names what would retire it, which turns API drift into a tracked backlog rather than a surprise migration.
LessonWrite the configuration you change most in the most widely implemented language available, and keep an explicit, shrinking list of where you could not.
Shown on views05 13 28
ADR-15

The rollout controller owns traffic weights during a rollout, and judges the share the mesh reports

Accepted

Who owns the traffic weights while a release is rolling out?

Context
The candidates are the mesh API as system of record with the pipeline writing to it, a rollout controller that owns and reconciles weights, or the CD system writing weights directly. Each puts the authority to roll back somewhere different. During a partial failure, if two systems both believe they own a weight, the rollback races the rollout. The requirement also asks that the delivery system read effective weights from the mesh rather than assume its intent took effect.
Decision
Argo Rollouts owns the weights of a route for the duration of a rollout. It writes weight steps to the HTTPRoute, runs analysis against Thanos on the canary's reported share, error ratio and latency relative to stable in the same window, and aborts automatically on breach. A human stops a release by aborting the Rollout, never by editing the route. Outside a rollout, the routing repository is the owner.
How it works on-premise
The Rollout object references the HTTPRoute through the Gateway API traffic-router plugin. Admission guards deny updates to a route carrying an active-rollout annotation from any identity other than the Argo Rollouts controller's service account. Analysis templates are shared from the platform: applied share within ±20% of target, 5xx ratio no worse than 1.5× stable, p99 no worse than 1.3× stable, with a minimum request count per step.
Options weighed
  • ChosenRollout controller owns weights; analysis reads mesh telemetry: One writer during a rollout, an abort that is fast and in one place, and a judgement based on what the mesh did.
  • RejectedCD pipeline writes weights directly: No extra controller. A pipeline job that dies mid-rollout leaves weights with no owner, and its abort path is a second job.
  • RejectedGit as system of record for every weight step: Perfect audit trail. A rollback becomes a commit plus a sync, which is minutes when the requirement asks for under 60 seconds.
Consequences
What it buys
  • Rollback and rollout cannot race, because only one identity may write the route
  • An abort is one command against one object and takes effect in seconds
  • A weight change that never landed is detected before a canary is judged
What it costs
  • During a rollout the live route differs from Git; the route's intent sha reflects the Rollout object, not a commit
  • Argo Rollouts is on the path of every release, so its availability matters on release days
  • Low-traffic services need longer steps to reach the minimum request count
Choose differently when
Where releases are rare and manual and there is no SLO-driven analysis, weights written through Git with a normal review are simpler, and the 60-second rollback target should be renegotiated rather than faked.
Why it holds up over time
The principle, one writer per mutable field and judgement on observed state, is independent of Argo Rollouts. Any replacement controller inherits the same admission rule and the same analysis contract, and the handover between Git ownership and rollout ownership is an annotation any tool can respect.
LessonGive every mutable field exactly one writer at any moment, and judge an action by what the system reports happened, not by what you asked for.
Shown on views05 18 28
ADR-16

Revert pins a previously rendered bundle, and is the one operation allowed ahead of Git

Accepted

How is reverting a mesh change made the fastest path in the system?

Context
In a GitOps flow, undoing a change means a revert commit, a review, a render and a sync, which is minutes. The requirement asks for revert to a previously applied configuration as a single operation and for revert to be the fastest path, with automated rollback inside 60 seconds of a breach. A revert that waits on the same pipeline as the change it undoes is as slow as the change.
Decision
Every merged change is rendered once into a signed OCI bundle in Harbor, and each cluster records which bundle digest it has applied. Revert is a single command (or an automatic action from a wave gate) that pins the affected clusters to the previous digest: it suspends Argo CD's automatic sync for that application, applies the stored bundle directly, and opens the Git revert commit afterwards. Once the revert commit merges, sync resumes and the pin is released.
How it works on-premise
Bundles are Kubernetes manifests packaged with ORAS, signed with cosign using a key in OpenBao, and verified by admission before apply. The revert command is a small controller with its own identity whose only permissions are to suspend sync and apply a verified bundle to mesh resource kinds. Every revert writes an audit event with the from and to digests; a pin older than two hours pages the owning team.
Options weighed
  • ChosenPin a pre-rendered signed bundle, then catch Git up: Seconds, not minutes; nothing is recompiled; Git remains the record once the revert commit lands.
  • RejectedGit revert and normal sync: Pure GitOps. Slower than the change it undoes, and depends on the review and sync path being healthy during an incident.
  • Rejectedkubectl apply of the previous version by hand: Fast. Unverified, unaudited, and Argo CD reapplies the bad version at its next sync.
Consequences
What it buys
  • Rollback meets 60 seconds without depending on CI, review or rendering
  • The reverted state is exactly what was running before, verified by signature
  • Argo CD cannot fight the revert, because its sync is suspended for the duration
What it costs
  • For a short window the cluster is ahead of Git, which is a deliberate and alerted exception to GitOps
  • Bundle storage and signing add a component to the change path
  • A revert of a change that also altered non-mesh resources reverts only the mesh kinds
Choose differently when
For platforms whose changes are rare and whose rollback target is measured in tens of minutes, a plain Git revert is simpler and keeps the invariant that the cluster never leads Git.
Why it holds up over time
Content-addressed, signed artefacts are how software supply chains are moving in general, so the bundle store is not a mesh-specific invention. The rule that the undo path must not share dependencies with the do path applies to whatever replaces Argo CD.
LessonMake undo independent of the machinery that produced the mistake, and keep what you would undo to already built and signed.
Shown on views14 23 12
ADR-17

Timeouts are mandatory, retries are budgeted, and deadlines travel with the call

Accepted

How do proxy retries hide a transient failure without turning a slow dependency into a fleet-wide outage?

Context
Retries in a proxy are a shared amplifier. Three attempts per call triple the load on a dependency that is already failing, and nested retries multiply: three hops each retrying three times send 27 requests to the bottom of the stack. The requirement asks for a per-route timeout with no route allowed without one, retries only for declared idempotent methods and conditions with bounded attempts, per-try timeout and jitter, a retry budget per destination, and deadlines that bound nested retries.
Decision
Every route declares a timeout; admission rejects a route without one, and the platform default applied by the generator is 2 s. Retries apply only to GET, HEAD and methods a service declares idempotent, on connection failure, reset and 503, with at most three attempts, a per-try timeout and 25 ms base backoff with full jitter. Each destination has a retry budget of 20% of active requests. The caller's remaining deadline travels as a header; the inbound proxy caps its route timeout by it, and no retry starts if the remaining deadline cannot cover a per-try timeout.
How it works on-premise
Timeouts and retry policy are set on HTTPRoute (or DestinationRule where the standard lacks the field). Envoy's retry_budget in cluster circuit breakers and the router's respect_expected_rq_timeout option are not exposed by Istio's API on every release, so they are applied by one EnvoyFilter generated per revision, owned by the platform team and exercised by the upgrade conformance suite. Applications forward the deadline header alongside trace context through the same OpenTelemetry propagator configuration.
Options weighed
  • ChosenMandatory timeouts, budgeted retries, propagated deadlines: Bounds amplification at every level: per call, per destination and per call chain.
  • RejectedAttempt counts only: Available through the standard API today. Bounds one call and does nothing about a thousand callers retrying together.
  • RejectedNo proxy retries; applications decide: No amplification from the mesh. Gives up the requirement's transparent handling of transient failure.
Consequences
What it buys
  • A widespread failure increases load on the failing service by at most a fifth
  • Nested calls cannot multiply retries past the original caller's deadline
  • No route in the mesh can wait forever
What it costs
  • Two features depend on an EnvoyFilter, which is a documented upgrade risk
  • Deadline propagation needs every application to forward one header
  • Some legitimate slow endpoints need explicit longer timeouts, which teams must declare
Choose differently when
Where call chains are shallow (one or two hops) and traffic is low, attempt counts with jittered backoff are enough, and the EnvoyFilter is not worth its upgrade risk.
Why it holds up over time
Budgets and deadlines are properties of the traffic, not of Istio; gRPC, Envoy and every mature RPC stack implement them. When the standard API exposes the budget, the EnvoyFilter is deleted and nothing about the behaviour changes.
LessonBound retries by a fraction of real traffic and by the caller's deadline, not only by an attempt count, or the mesh becomes the amplifier in every outage.
Shown on views11 20 31
ADR-18

Outlier ejection and load shedding at the proxy; locality routing with a declared spillover threshold

Accepted

How does the mesh handle a bad instance, an overloaded destination and a failing zone?

Context
A single bad pod should leave the load-balancing set quickly and come back cautiously. An overloaded destination should reject fast instead of queueing until every caller times out. Nearest-zone routing saves latency and cross-zone cost, but the point at which traffic spills to another zone or cluster is a cost and blast-radius decision that defaults hide. The requirement asks for ejection with bounded, escalating return, per-destination concurrency and pending limits, and a declared spillover threshold.
Decision
Every destination has outlier detection: eject after five consecutive 5xx or gateway errors, base ejection 30 s multiplied by the number of prior ejections up to 5 minutes, and at most 50% of endpoints ejected. Connection pools cap connections, pending requests and active requests per destination, with pending requests limited so the callee's proxy sheds with a fast 503 marked as overflow. Locality-aware routing prefers the caller's room, then its site, then remote clusters, and spills over when fewer than 70% of the preferred locality's endpoints are healthy, declared per destination.
How it works on-premise
Settings live in DestinationRules generated from the service manifest with platform defaults and per-service overrides. Locality comes from standard topology labels on nodes (room as zone, site as region). Envoy's overprovisioning factor is set per destination so that the 70% threshold is the number written in the manifest, not an implicit 1.4 factor. Spillover requests are counted by source and destination locality for the cost report.
Options weighed
  • ChosenProxy-side ejection, shedding and declared locality spillover: Uniform behaviour for every service, visible thresholds, and overload refused where it is cheapest.
  • RejectedClient libraries with circuit breakers: Rich per-call context. Different behaviour per language and per library version across 250 teams.
  • RejectedRound-robin across all localities: Even load. Pays cross-room latency and cost on every request and spreads a failing zone's errors everywhere.
Consequences
What it buys
  • Bad pods leave rotation in seconds and repeat offenders stay out longer
  • Overload produces fast, attributed failures instead of timeouts
  • Cross-zone and cross-cluster traffic happens only when a declared threshold is crossed
What it costs
  • Aggressive ejection on a small pool can remove healthy capacity; the 50% cap is the guard
  • Limits set too low shed traffic a service could have handled; they need load testing per service
  • Locality routing concentrates load in one room when callers are unevenly placed
Choose differently when
For small services with two or three pods, disable percentage-based ejection and rely on health checks, because ejecting one pod removes a third of capacity.
Why it holds up over time
Ejection, concurrency limits and locality are standard load-balancer behaviour in every proxy generation. Declaring thresholds per destination, in reviewed data, means they survive a change of proxy and remain visible to whoever inherits the service.
LessonMake every threshold that decides where traffic goes a declared number, because an implicit default is a decision nobody remembers making.
Shown on views11 19 20

Multi-cluster and networkHow traffic crosses a cluster or a site, how trust between clusters is declared, and what the network under the mesh provides.

ADR-19

Flat pod networking inside a site; east-west gateways between sites

Accepted

How does traffic cross a cluster boundary?

Context
A flat network, with every pod IP routable and remote endpoints in local discovery, gives the shortest path and the simplest routing, and requires non-overlapping address space and a posture where one cluster's compromise reaches every pod at L3. East-west gateways expose each cluster through one hop, add latency and a component, contain blast radius, and work across networks never designed to be joined. Here the network inside each site is under the platform team's control; links between sites are not.
Decision
Clusters within one site share a routable pod network with non-overlapping CIDRs, and calls between them go pod to pod with mTLS. Calls between sites go through east-west gateways that route on SNI in passthrough mode, so TLS from the caller's proxy reaches the destination proxy intact and the gateway holds no workload identity. The MVP is single-site, which matches the requirement's single-region MVP.
How it works on-premise
Each site allocates a /14 of pod addresses and assigns each cluster a non-overlapping block, advertised by Cilium over BGP to the site's leaf switches. Cilium network policy denies cross-cluster traffic by default except between meshed namespaces. East-west gateways run as Envoy in Istio's AUTO_PASSTHROUGH mode on a dedicated gateway node pool, behind a BGP-announced virtual address with ECMP across gateway pods.
Options weighed
  • ChosenFlat inside a site, gateways between sites: Shortest path where the network is ours and failure domains are shared anyway; containment where links cross sites.
  • RejectedFlat everywhere: Simplest routing. Needs routed pod addresses across sites and lets a compromise in one site reach every pod in another.
  • RejectedGateways everywhere, even inside a site: Uniform model. Adds a hop and a component to traffic that already shares switches, power and failure domain.
Consequences
What it buys
  • Most cross-cluster traffic, which stays inside a site, pays no extra hop
  • Cross-site traffic is contained to named gateway addresses that firewalls and capacity plans can see
  • Caller identity reaches the destination intact across sites
What it costs
  • Two cross-cluster models to operate and debug
  • Pod address planning per site is a prerequisite and hard to change later
  • Gateways are capacity to size for cross-site failover, not for normal traffic
Choose differently when
If the network inside a site is owned by another organisation, or clusters inside a site belong to different security zones (a PCI enclave, for example), put gateways between those clusters too.
Why it holds up over time
The split follows ownership of the network, which changes slowly. Adding a site adds gateways without touching existing sites, and SNI passthrough preserves end-to-end identity regardless of which proxy implementation runs in the gateway.
LessonJoin networks directly only where you own both ends and share a failure domain; everywhere else, make the crossing a named, narrow, identity-transparent hop.
Shown on views19 22 10
ADR-20

Cross-cluster trust and service export are declared, never default

Accepted

When does one cluster accept another cluster's identities and see its services?

Context
Multi-cluster meshes commonly merge every service from every cluster into every other cluster's discovery, and accept every identity chained to the shared root. That is convenient and means a cluster's full service list and every workload identity are trusted everywhere without anyone having decided so. The requirement asks for trust accepted only by declared federation, discovery partitioned per cluster, and replication of only what federation declares.
Decision
Each service that should be reachable from other clusters is exported explicitly in the routing repository, naming the clusters allowed to import it. Each cluster's authorisation policies accept identities from other trust domains only where the security repository lists them for that destination. A cluster with no declarations sees and trusts only itself, even though it shares the mesh root.
How it works on-premise
Exports are a label on the Service plus an entry in a federation manifest that the routing generator turns into istiod discovery selectors and remote-secret scope for peer clusters. Trust is expressed as principals in AuthorizationPolicy including the peer's trust domain (for example c12.mesh.corp/ns/shop/sa/checkout), so an unlisted cluster's identity fails authorisation even with a valid chain.
Options weighed
  • ChosenExplicit export and explicit trust per destination: Discovery and trust grow only by reviewed decisions, and each is visible in Git.
  • RejectedMerge all services, trust the whole root: Zero configuration. Every cluster's compromise and every cluster's naming mistake is shared.
  • Right elsewhereSeparate meshes joined by gateways with translation: Right between organisations or companies. Loses end-to-end identity, which this design needs inside one organisation.
Consequences
What it buys
  • Losing or compromising a cluster exposes only what it exported and whom it was trusted by
  • Per-cluster discovery stays small, which keeps istiod and proxy memory bounded
  • A reviewer can answer 'who can call this from another cluster' from Git
What it costs
  • Every cross-cluster dependency needs two declarations, export and trust
  • Failover to a cluster that did not export a service does not happen, which must be caught in review
  • Discovery selectors and remote secrets need tooling to stay consistent with the manifest
Choose differently when
In a small mesh of identical, interchangeable clusters (one application deployed to three clusters for capacity), a namespace-level export of everything is simpler and the review burden buys little.
Why it holds up over time
Declared federation is the SPIFFE model and the direction multi-cluster service APIs in Kubernetes take. The manifest is plain data, so it can drive a future mesh's multi-cluster mechanism without re-deciding what is shared.
LessonSharing a root is a cryptographic convenience; trust should still be something each destination grants on purpose.
Shown on views19 09 12
ADR-21

Cilium for pod networking, BGP-announced gateway addresses and a second enforcement plane

Accepted

What provides the network under the mesh on hardware without a cloud load balancer?

Context
The requirement's cloud stack puts network load balancers in front of east-west gateways. On-premise, the equivalent must come from the cluster itself or from physical load balancers. The mesh also needs a network layer that can block traffic independently of the mesh during an incident (ADR-04, ADR-13), and routable pod addresses within a site (ADR-19).
Decision
Cilium is the CNI on every cluster, in eBPF mode with kube-proxy replacement. Its BGP control plane peers with the site's leaf switches to advertise pod CIDRs and the virtual addresses of east-west and egress gateways, allocated by its load-balancer IPAM, with ECMP across gateway pods. CiliumNetworkPolicy provides default-deny between namespaces at L3/L4 as the second enforcement plane.
How it works on-premise
Each node peers with two leaf switches. Gateway virtual addresses are announced only from nodes running a ready gateway pod, so a failed node withdraws its route in seconds. Cilium's own L7 and mutual-authentication features are left off to keep a single L7 enforcement point (Istio) and avoid two proxies on the same connection.
Options weighed
  • ChosenCilium with BGP and LB IPAM: CNI, virtual addresses and L3/L4 policy from one component with its own agent per node.
  • RejectedCalico with MetalLB: Mature and widely run. Two components for what Cilium does in one, with no eBPF service load balancing.
  • RejectedHardware load balancers in front of gateways: Familiar to network teams. A ticket-driven change for every new gateway address and a device outside the cluster's failure domain.
Consequences
What it buys
  • Gateway addresses follow healthy pods automatically, without a separate device
  • An emergency block is available that does not depend on istiod
  • kube-proxy's iptables scaling limits are removed at 2,000 pods per cluster
What it costs
  • Network and platform teams share responsibility for BGP sessions
  • Two policy systems (Cilium and Istio) must be kept consistent
  • Cilium's kernel requirements constrain the node operating system
Choose differently when
If the network team requires all load balancing on dedicated appliances with their own change process, keep appliances for gateway addresses and use Cilium only as CNI and policy engine.
Why it holds up over time
BGP is the most stable protocol in the building. Cilium's role here is deliberately narrow (routes, addresses, L3/L4 policy), so replacing it later with another eBPF or BGP-capable CNI changes no mesh configuration.
LessonUse the network layer for what it is best at, reachability and coarse policy, and keep it independent of the mesh so each can protect against the other's failure.
Shown on views07 08 22

Change and governanceWhere intent lives, who may change which part, how a change and an upgrade reach 40,000 proxies, and what is recorded.

ADR-22

Intent in three repositories, with separation of duties enforced by repository and by Argo CD project

Accepted

How is it enforced, not just agreed, that people who change routing cannot change authorisation or identity?

Context
The requirement asks for separation of duties between routing and authorisation or identity changes, for intent to be versioned immutably and traceable to what each proxy runs, and for strong consistency with full history. Review conventions such as code owners are bypassable by repository administrators and, in some Git services, are paid features. Kubernetes RBAC alone does not know which team a change came from.
Decision
Mesh intent lives in three repositories on Forgejo: routing (service teams and the platform team), security (authorisation, peer authentication, trust declarations; security team), and identity (SPIRE templates, intermediates, bundles; PKI owners). Commits are signed. Each repository is applied by its own Argo CD project whose allowed resource kinds match that repository, so the routing project cannot create an AuthorizationPolicy even if one is committed there.
How it works on-premise
Forgejo runs on PostgreSQL with a synchronous standby in a second room, with read-only push mirrors at the other two sites. Branch protection requires signed commits and two approvals from the repository's owner team. Argo CD AppProjects list namespaceResourceWhitelist per repository; admission guards reject mesh objects whose manager is not the expected Argo CD project. Every applied object carries the commit sha as an annotation.
Options weighed
  • ChosenThree repositories, three Argo CD projects with kind allowlists: Two independent enforcement layers, neither of which depends on reviewers noticing a file path.
  • RejectedOne repository with code-owner rules: Simpler cross-cutting changes. Enforcement depends on the Git service's paid features and on administrators not bypassing them.
  • RejectedDirect kubectl with RBAC per team: No GitOps machinery. No review, no signed history, and no link from a proxy's configuration to an intent version.
Consequences
What it buys
  • A routing change cannot alter who may call a service, by construction
  • Every applied object is traceable to a signed commit and its approvers
  • The auditor's question 'who changed this policy' is a Git log query
What it costs
  • Changes that need a route and a policy together need two merges in order
  • Three repositories and three projects to maintain
  • A security-team bottleneck on authorisation changes unless reviews are staffed
Choose differently when
A small organisation where the same five people run routing and security gains nothing from the split, and one repository with protected paths is enough.
Why it holds up over time
The split follows organisational roles, which change more slowly than tools. Moving from Forgejo to another Git service or from Argo CD to another reconciler preserves the three-repository boundary and its kind allowlists.
LessonEnforce separation of duties where changes are applied, not only where they are reviewed.
Shown on views13 29 04
ADR-23

Argo CD per site, and every mesh change rolls out in cluster waves behind health gates

Accepted

How does a valid-looking change that breaks routing avoid breaking it everywhere at once?

Context
A configuration that passes every schema and policy check can still break routing mesh-wide, for example a default timeout that is too short for one heavy path. The requirement asks for validation, staged waves with per-wave health gates, and automatic halt and revert on error-rate breach, and for no cross-region dependency for in-region configuration.
Decision
Each site runs its own Argo CD managing only that site's clusters, reading from the local Forgejo mirror. Mesh applications sync in waves: wave 0 is a canary cluster carrying synthetic and a slice of real traffic, wave 1 is one cluster per site, wave 2 is the remaining clusters site by site. A gate between waves waits a fixed soak and checks mesh-attributed error ratio, NACK count and proxies still on the previous sha. On breach it halts and triggers the revert in ADR-16.
How it works on-premise
Waves are ApplicationSets with cluster generators ordered by a wave label; the gate is a small controller with read access to Thanos and permission only to advance a wave label or call the revert controller. Argo CD instances run in the site's management cluster with three replicas. Emergency denies are a labelled change type that syncs all clusters without waves.
Options weighed
  • ChosenPer-site Argo CD, cluster waves, automated gates: A bad change reaches one cluster first and is halted by signals the mesh already emits.
  • RejectedCentral Argo CD for all sites: One place to look. A site partition stops changes in the other sites and puts cross-site latency in every sync.
  • RejectedSync everywhere at once, rely on validation: Fastest convergence. Validation cannot catch a change that is valid and wrong.
Consequences
What it buys
  • A valid but wrong change affects one cluster before the gate halts it
  • Sites change independently during a partition
  • Gates use mesh signals, so unrelated application incidents do not block changes
What it costs
  • A full rollout takes about 45 minutes instead of seconds
  • The canary cluster must carry enough real traffic for its signals to mean something
  • Emergency changes bypass waves and depend on their own review after the fact
Choose differently when
With two or three clusters, waves add little over a canary namespace inside one cluster; stage by namespace rather than by cluster.
Why it holds up over time
Progressive rollout of configuration is how large operators have converged on handling configuration outages, and it is independent of the reconciler. The wave labels and gate contract are plain data any delivery tool can respect.
LessonValidation proves a change is well formed; only staged exposure proves it is right. Stage configuration the way you stage code.
Shown on views23 14 22
ADR-24

Proxy configuration is scoped to declared dependencies and distributed as deltas

Accepted

What is the unit of configuration distribution?

Context
Full snapshots per proxy on every change are simple and idempotent, and at 40,000 proxies make one route change a fleet-wide push. Deltas cut that by orders of magnitude and introduce sequencing, resynchronisation and silent divergence. Scoping each proxy to the destinations its workload calls helps both and needs a dependency graph, which, if inferred, is wrong exactly when a service starts calling something new. Unscoped, every proxy's memory grows with the whole mesh.
Decision
Scope every proxy with an Istio Sidecar resource. In the MVP the scope is the workload's own namespace plus namespaces it declares. In Phase 2 the scope is generated from per-service dependency declarations in the routing repository. Configuration is distributed as delta xDS. A call to an undeclared destination fails with a distinct response flag, is attributed to mesh configuration, and alerts the owning team with the missing declaration.
How it works on-premise
The routing generator turns each service manifest's dependsOn list into a Sidecar egress host list. A CI check compares declarations against the last 7 days of access logs and warns on declared-but-unused and on observed-but-undeclared destinations. The drift exporter forces a proxy to resubscribe if its acknowledged sha lags intent by more than 60 seconds, which bounds the delta divergence class of bug.
Options weighed
  • ChosenDeclared scope, delta distribution, drift-bounded: Memory tracks what a workload calls, pushes cost what they touch, and divergence is detected within a minute.
  • RejectedFull mesh configuration to every proxy: Nothing to declare and nothing to go wrong. Memory and push cost grow with the fleet and break the budget well before 40,000 proxies.
  • RejectedScope inferred from observed traffic: No declarations to maintain. Refuses a new dependency on the first call, which is the moment nobody is watching.
  • DeferredFull snapshots, scoped: Idempotent and easier to reason about. Kept as the fallback if delta distribution misbehaves on the chosen release.
Consequences
What it buys
  • Per-proxy memory stays roughly constant as the mesh grows
  • A one-service change reaches only the proxies that call that service
  • Dependency declarations double as documentation and as input to authorisation reviews
What it costs
  • Teams must declare dependencies, and a missing declaration refuses calls in scoped namespaces
  • Delta distribution adds a class of bug where a proxy silently lags intent
  • The generator and the log comparison are platform code to maintain
Choose differently when
Below a few hundred services, full configuration to every proxy fits the memory budget, and the declaration burden is not worth it.
Why it holds up over time
Scope by declared dependency is independent of the mesh; any xDS control plane, and any per-node data plane, benefits from the same graph. The graph is also the input to future work that has nothing to do with proxies: blast-radius analysis, ownership and change impact.
LessonMake what a service depends on a declared, reviewed fact, and let configuration size, policy review and impact analysis all read from it.
Shown on views14 26 27
ADR-25

Admission guards are the second line: OPA Gatekeeper and ValidatingAdmissionPolicy

Accepted

What stops an invalid or unsafe mesh object that never went through CI?

Context
CI checks run on changes that go through Git. Anything applied by kubectl, by a controller, or by a restored backup skips them. Istio accepts many configurations that are valid and unsafe: a route with no timeout, retries on POST, a VirtualService racing a Gateway API route, a route edited while a rollout owns it. The requirement's rules must hold regardless of how an object arrived.
Decision
Every rule that CI enforces on mesh objects is also enforced at admission in every cluster. Structural rules that need no external data (timeouts present, retry methods idempotent, sidecar present in strict namespaces) are ValidatingAdmissionPolicies evaluated inside the API server. Rules that need cluster context (an active rollout owns this route, this object's manager matches its repository) are OPA Gatekeeper constraints.
How it works on-premise
Constraint templates and admission policies live in the identity-and-platform repository and are applied in wave 0 first. Gatekeeper runs three replicas with failurePolicy Fail for mesh kinds in strict namespaces and Ignore elsewhere, and its audit mode reports existing objects that violate a new constraint before the constraint enforces. The same Rego is used by conftest in CI, so the two lines cannot drift.
Options weighed
  • ChosenCEL admission policies plus Gatekeeper, same rules as CI: No path around the rules, and the simplest rules have no webhook to fail.
  • RejectedCI checks only: One place to maintain. Anything applied outside Git is unchecked, including during incidents when rules matter most.
  • RejectedKyverno: YAML policies many teams find easier than Rego. The organisation's security team already writes Rego for conftest, and one language for both lines avoids drift.
Consequences
What it buys
  • The mesh's safety rules hold for objects from any source
  • Most rules run in the API server with no additional availability dependency
  • Audit mode shows the impact of a new rule before it blocks anything
What it costs
  • Gatekeeper is another webhook whose availability matters for strict namespaces
  • Rules exist in two forms (CEL and Rego) that must stay consistent
  • Overly strict rules block emergency changes unless an audited bypass exists
Choose differently when
If every change to every cluster provably goes through one pipeline, with cluster-admin credentials held only by that pipeline, admission guards duplicate CI and can be reduced to the rules about pods.
Why it holds up over time
Admission control is part of Kubernetes, and CEL admission policies are the direction the project has taken to reduce webhook dependencies. Rules written against resource fields survive changes of the tools that create those resources.
LessonEnforce invariants at the last point every change must pass through, not only at the point most changes pass through.
Shown on views02 07 14
ADR-26

Proxy fleet upgrades by control-plane revision, never in place

Accepted

How are 40,000 proxies upgraded with per-wave verification and an abort that leaves a working fleet?

Context
Upgrading istiod in place means every proxy in the cluster immediately talks to a newer control plane, and rolling back means downgrading a control plane that proxies of a newer version may already be using, which upstream does not support. The requirement asks for upgrades in declared waves with health verification, an abort that leaves a mixed but working version set, and a skew window of at least one minor version in each direction.
Decision
A new Istio minor version is installed as a new revision beside the current one. Namespaces move by changing their revision tag, from low-risk to tier-1 in three waves, and their pods restart onto the new proxy. Each proxy connects only to the istiod of its own revision, so a proxy never talks to an older control plane and skew in both directions is held by construction. Abort is retagging back and restarting. The old revision is removed only when no proxy uses it.
How it works on-premise
Revision tags (canary, stable) are managed with istioctl tag in the platform repository, applied through Argo CD. Rolling restarts are rate-limited to the SPIRE issuance headroom and gated by PodDisruptionBudgets. The conformance suite, including the retry-budget EnvoyFilter and dry-run policies, runs against each new revision in wave 0 before any namespace moves. The platform upgrades at least quarterly to stay inside upstream support.
Options weighed
  • ChosenRevision-based canary upgrades: Both versions serve in parallel, abort is a retag, and no unsupported version pairing ever exists.
  • RejectedIn-place control-plane upgrade, then proxy restarts: One istiod at a time. Every proxy meets the new control plane at once, and rollback pairs newer proxies with an older control plane.
  • RejectedBlue-green whole clusters: Cleanest isolation. Needs double capacity per cluster for every upgrade, four times a year.
Consequences
What it buys
  • A problem in a new version affects only the namespaces moved so far
  • The fleet can sit mixed for weeks if a wave needs investigation
  • Upgrades are frequent and small, which keeps each one cheap
What it costs
  • Two control planes run per cluster during an upgrade, doubling istiod resources temporarily
  • Every upgrade restarts every meshed pod in the cluster
  • Teams see pod restarts they did not initiate, four times a year
Choose differently when
For patch releases within a minor version, in-place upgrades with a rolling restart are simpler and the skew risk is negligible; reserve revisions for minor versions.
Why it holds up over time
Side-by-side versions with traffic moved in waves is the general pattern for upgrading anything on a request path. The quarterly cadence prevents the multi-version jump that turns an upgrade into a project, which is how meshes are most often abandoned.
LessonUpgrade by running old and new side by side and moving tenants, and upgrade often enough that each move is small.
Shown on views24 31 23
ADR-27

An append-only audit trail on Object Lock storage, independent of the SIEM

Accepted

Where is the record of who changed what, and who got which certificate, kept so that it cannot be edited?

Context
The requirement asks that every configuration change, policy change and certificate issuance be recorded in an append-only trail with actor, before and after state and time, retained for 13 months. Git history can be rewritten by an administrator, Kubernetes audit logs rotate on the API server, and a SIEM's retention is set by another team's budget.
Decision
Three sources feed one trail: Git commits with signatures and approvers, Kubernetes audit events for mesh resource kinds (including before and after objects), and SPIRE issuance records. A collector writes them hourly as compressed files to a Ceph RGW bucket with S3 Object Lock in compliance mode and a 13-month retention. The SIEM receives a copy for correlation but is not the record.
How it works on-premise
API server audit policy logs RequestResponse for mesh CRDs and metadata for everything else. The OpenTelemetry Collector routes audit and issuance streams to both ClickHouse (for query) and the Object Lock bucket (for evidence). A daily job verifies the bucket against ClickHouse counts and the Git log and alerts on gaps. Forgejo's own backups also go to an Object Lock bucket.
Options weighed
  • ChosenObject Lock bucket as record, SIEM as copy: Immutable by storage policy, retention set by this platform's requirement, queryable through ClickHouse.
  • RejectedSIEM as the record: Already exists and is searched by the security team. Retention and immutability are outside this platform's control.
  • RejectedGit history alone: Captures intent. Misses changes applied outside Git and says nothing about certificate issuance.
Consequences
What it buys
  • The audit trail cannot be altered, even by the platform's administrators, within retention
  • Changes applied outside Git appear in the same trail as those that went through it
  • Retention is guaranteed by storage policy, not by a scheduled job
What it costs
  • Compliance-mode Object Lock cannot be shortened, so a wrong retention setting costs storage for 13 months
  • API server audit at RequestResponse level for mesh kinds adds load and volume
  • A reconciliation job is needed to detect gaps from the loss-tolerant telemetry path
Choose differently when
If the organisation's SIEM already provides immutable retention beyond 13 months with guarantees the audit function accepts, write directly to it and drop the separate bucket.
Why it holds up over time
S3 Object Lock is an interface implemented by Ceph, MinIO and every major object store, so the evidence store can move without changing its guarantee. The trail records facts (commits, API events, certificates) rather than tool-specific logs.
LessonKeep evidence where its immutability is guaranteed by the storage, and treat every search tool as a copy of it.
Shown on views12 13 03

Observability and costWhat every hop reports, how a failure is attributed, how drift is seen, and how the mesh's own cost is kept inside a number.

ADR-28

Prometheus per cluster with Thanos for global query and history; labels pruned at the proxy

Accepted

How are per-hop metrics collected from 40,000 proxies without the metrics system becoming the most expensive part of the mesh?

Context
The requirement asks for rate, errors and duration for every hop by source identity, destination service, subset and response class, retained 15 days raw and 13 months rolled up. Envoy emits thousands of series per proxy by default, and per-pod labels multiply them by 40,000. Rollout analysis and wave gates read these metrics, so the system must answer within seconds during an incident.
Decision
Each cluster runs Prometheus in HA pairs scraping its proxies. Istio's Telemetry API reduces standard metrics to the labels the requirement names, and Envoy's own stats are limited to an allowlist. Recording rules aggregate to service-to-service series; per-pod labels are dropped at scrape. Thanos sidecars upload blocks to Ceph RGW; Thanos compactor downsamples to 5-minute and 1-hour resolution for 13 months; Thanos Query at each site serves the site, and a global query layer serves cross-site views.
How it works on-premise
Prometheus runs on the platform node pool with local SSDs for the 2-hour head block. Thanos stores and compactors run in each site's management cluster against a site-local Ceph RGW bucket. Rollout analysis and gates query the site-local Thanos, never the global layer, so a site partition does not blind them. Alertmanager runs per site with a cross-site mesh for deduplication.
Options weighed
  • ChosenPrometheus per cluster with Thanos: Collection stays local to the failure domain, history is cheap on object storage, and the query path for gates is site-local.
  • RejectedVictoriaMetrics cluster: More efficient storage and simpler scaling. Clustering and some enterprise features differ between editions, and the team already runs Prometheus.
  • RejectedGrafana Mimir: Horizontally scalable and multi-tenant. More moving parts than a platform team of this size needs for 20 clusters.
Consequences
What it buys
  • Metrics collection survives a site partition and a telemetry backend outage
  • Thirteen months of history cost object storage, not SSD
  • Label pruning keeps series count proportional to service pairs, not pods
What it costs
  • Per-pod debugging needs a temporary, scoped label override
  • Thanos adds components (sidecar, store, compactor, query) to operate
  • Cross-site queries are slower and fail during partitions
Choose differently when
If per-series cost becomes the limiting factor at the design ceiling, move storage to VictoriaMetrics behind the same Prometheus query API; dashboards and gates do not change.
Why it holds up over time
PromQL and the Prometheus exposition format are the most stable interfaces in observability, and OpenTelemetry metrics map onto them. Choosing the label set deliberately, in the Telemetry API, is what keeps cost stable as the fleet grows, whichever backend stores it.
LessonDecide which labels a metric may carry before you decide where to store it; cardinality, not storage, sets the cost of mesh telemetry.
Shown on views15 25 07
ADR-29

Access logs to ClickHouse and traces to Tempo, sampled per namespace, never blocking a request

Accepted

How are per-hop access logs and spans kept useful without being the runaway cost of the mesh?

Context
At 400,000 requests a second through two proxies each, full access logging is about 70 billion lines a day. The requirement asks for access logs with peer identity, route, subset, retry count and terminating condition for 30 days, a span per proxy hop for 7 days, sampling and cardinality configurable per namespace with a declared default, no payloads or authorisation header values by default, and bounded, drop-oldest buffering that never blocks a request.
Decision
Envoy sends access logs over gRPC to a node-local OpenTelemetry Collector, which forwards to ClickHouse. Default capture is every denial and every 4xx or 5xx, plus 1% of successes; a namespace raises it with a reviewed change with an expiry date. Spans are emitted per hop with head-based sampling at 1% by default, always sampled when the caller has already sampled, and stored in Grafana Tempo on Ceph RGW. Collectors use bounded queues that drop the oldest data and count drops.
How it works on-premise
ClickHouse runs as a replicated cluster per site with a table partitioned by day, a 30-day TTL, and a projection on peer identity for incident queries. Header values are excluded by the access-log format; the log carries header names only where declared. Tempo uses Ceph RGW for blocks with a 7-day retention. Collector drop counters are scraped by Prometheus and alert above 0.1% sustained.
Options weighed
  • ChosenClickHouse for logs, Tempo for traces, per-namespace sampling: Columnar storage suits high-volume structured logs; object-storage traces are cheap; the default keeps every error.
  • RejectedOpenSearch for logs: Full-text search many teams know. Indexing cost at this volume is several times ClickHouse's for structured fields.
  • RejectedJaeger for traces: CNCF graduated and widely used. Needs a separate storage cluster where Tempo uses the object store already run for Thanos.
  • RejectedLog everything, sample nothing: Never misses a request. Makes telemetry the largest line in the mesh budget.
Consequences
What it buys
  • Every failure and denial is logged in full; successes are sampled
  • A slow or full telemetry backend never slows a request
  • Privacy defaults are enforced by the log format, not by convention
What it costs
  • A rare successful call may not be in the logs, which weakens the impact replay in ADR-12
  • Two storage systems for two signal types
  • Teams that need full success logging pay for it visibly and temporarily
Choose differently when
For a low-volume, highly regulated service (payments settlement), log every request unsampled in its own namespace policy with a longer retention, and account for it in that team's budget.
Why it holds up over time
OpenTelemetry is the collection standard, so the backends can change without touching 40,000 proxies. The sampling policy is data per namespace, which lets cost follow value as traffic grows rather than as a global compromise.
LessonKeep every error and a sample of success, make exceptions explicit and temporary, and never let the record of a request slow the request.
Shown on views15 25 26
ADR-30

Every failure signal carries its origin: application, mesh policy or configuration, or network

Accepted

When a call fails, how does anyone know whether to page the service team, the mesh team or the network team?

Context
A 503 can mean the application returned it, the callee's proxy shed load, no healthy upstream existed, an authorisation rule denied it, the route was missing, or a TLS handshake failed on a trust-bundle mismatch. Without distinguishing these, every mesh incident starts with a debate about whose fault it is, and the 99.99% mesh-attributable availability target cannot be measured. The requirement asks for that distinction in every failure signal.
Decision
Every request metric and access log carries a failure_origin label with four values (none, application, mesh, network) derived from Envoy's response flags and response code details. Mesh covers authorisation denial, no route, overflow, rate limit, and configuration or certificate errors; network covers connection failure, reset and upstream timeout without a response; application covers responses the upstream application produced. TLS failures are further split into trust-bundle skew and certificate validation, so skew is never reported as an authorisation problem.
How it works on-premise
The label is added through Istio's Telemetry API with a CEL expression over response.flags and response.code_details, and the mapping table lives in the platform repository with tests. The mesh availability SLO is computed from failure_origin="mesh" over all requests. Pages route by origin: mesh to the platform on-call, application to the owning team, network to the network on-call with the pair of localities.
Options weighed
  • ChosenOrigin label from response flags on every signal: Makes the mesh's own availability measurable and routes incidents to the right team from the first alert.
  • RejectedStatus codes and dashboards only: No extra labels. Every 503 looks the same, and the mesh SLO becomes a matter of opinion.
  • RejectedDistinct status codes for mesh failures: Visible to applications. Changes response semantics clients depend on and breaks retries keyed on 503.
Consequences
What it buys
  • The 0.01% mesh-attributable failure target is a query, not an argument
  • On-call routing is correct at the first page
  • Trust-bundle skew during a root rotation is visible as skew
What it costs
  • The mapping from flags to origin must be maintained as Envoy adds flags
  • One more label on request metrics (four values, so a bounded cost)
  • Some ambiguous cases, such as an upstream timeout caused by a slow application, are classified by rule and can be wrong
Choose differently when
A small mesh run by the same team that owns the services can skip the label and debug from access logs directly, because all pages go to the same people anyway.
Why it holds up over time
Every proxy exposes some form of termination reason; the stable part is the four-value classification and the SLO computed from it. If the proxy changes, only the mapping table does.
LessonA platform on the request path must be able to say, per failure, whether it caused it, or it will be blamed for everything and measured on nothing.
Shown on views15 20 31
ADR-31

Drift is observed per proxy, not inferred from sync status

Accepted

How does anyone know that the configuration a proxy runs is the configuration that was intended?

Context
Argo CD reporting Synced means objects are in etcd. istiod reporting a push means it sent something. Neither proves a proxy acknowledged it, and delta distribution adds a failure mode where a proxy diverges from intent with no error anywhere. The requirement asks that each proxy's effective configuration and version be exposed, so that drift is observed rather than inferred, and that any effective configuration be traceable to the intent version that produced it.
Decision
Every applied mesh object carries the commit sha that produced it. A drift exporter in each cluster reads istiod's per-proxy sync state (last sent and acknowledged nonce per resource type, proxy version) and the shas of the objects in each proxy's scope, and exports per proxy: the newest acknowledged sha, seconds behind intent, NACK count and proxy version. A proxy more than 60 seconds behind is forced to resubscribe; wave gates and rollout analysis read the same series.
How it works on-premise
The exporter is a small Go service using istiod's debug sync endpoint through an authenticated port, with the same data istioctl proxy-status shows, exported as Prometheus metrics with the proxy identity as a label only for proxies that are behind (to bound cardinality). Grafana shows drift per cluster, namespace and change sha.
Options weighed
  • ChosenPer-proxy acknowledged sha, exported continuously: Effective state is a metric; drift is a query; gates act on it automatically.
  • RejectedArgo CD sync status: Already available. Proves objects reached the API server, not that proxies run them.
  • Rejectedistioctl proxy-status on demand: Accurate. Only when someone thinks to run it, which is after users noticed.
Consequences
What it buys
  • A change is known to be live on a given share of proxies, with numbers
  • Silent divergence from delta distribution is detected within a minute
  • An auditor can trace a proxy's effective configuration to a signed commit
What it costs
  • A platform component that depends on istiod debug interfaces, which can change between releases
  • Forced resubscription adds control-plane load if many proxies lag at once
  • Cardinality control means per-proxy detail exists only for proxies that are behind
Choose differently when
If Istio exposes acknowledged configuration version per proxy as a first-class metric in a future release, replace the exporter with it and keep the gates and dashboards unchanged.
Why it holds up over time
The principle is to report effective state from the component that enforces it and to stamp intent with a version. Any xDS control plane has nonces and versions, so the exporter's contract survives a change of mesh.
LessonNever treat 'I sent it' as 'it is running'; stamp intent with a version and make the enforcing component report the version it holds.
Shown on views13 14 25
ADR-32

The mesh's cost is a declared budget, measured monthly and attributed to teams with OpenCost

Accepted

How is the cost of the mesh kept inside a number instead of being discovered at renewal?

Context
The mesh's cost is spread across every pod, every request and every byte of telemetry, which makes it easy to under-count and hard to reverse. Planning puts the idle floor of 40,000 proxies at about 400 vCPU, five times the per-request cost, and memory up to 4.8 TB. The requirement sets the mesh at no more than 8% of fleet CPU and 10% of fleet memory, reported monthly against actuals, attributed to owning teams, including telemetry volume and cross-zone traffic caused by routing.
Decision
OpenCost attributes the CPU and memory of every istio-proxy container, gateway and control-plane pod to namespaces and, through namespace labels, to teams. Telemetry volume per namespace comes from collector and ClickHouse metrics, and cross-zone and cross-cluster bytes from locality labels on request metrics. A monthly report compares the mesh's share of fleet CPU and memory with the budget and lists each team's share. Crossing 80% of either budget opens a review of scoping, sampling and the per-node L4 option.
How it works on-premise
Node costs are an internal rate per vCPU and GB of memory supplied by finance, loaded into OpenCost's custom pricing. Cross-site link cost is an internal rate per GB. Sidecar resource requests are set from measured usage percentiles per workload size class, not from defaults, and reviewed each quarter with the upgrade.
Options weighed
  • ChosenDeclared budget, OpenCost attribution, monthly report per team: The team that meshes a service sees its cost; the platform sees its trend against a number.
  • RejectedPlatform absorbs the cost centrally: Encourages adoption. Nobody who can change sampling or scoping sees the bill.
  • RejectedEstimate once at design time: Cheap. The idle floor and telemetry volume change with every release and every new service.
Consequences
What it buys
  • Cost regressions from a mesh upgrade or a sampling change are visible within a month
  • Teams have a reason to declare narrow dependencies and keep sampling at default
  • The per-node L4 decision in ADR-03 is made on measured numbers
What it costs
  • Attribution depends on accurate namespace-to-team labels
  • Internal rates are estimates finance must maintain
  • Chargeback can discourage teams from meshing unless the first year is shown but not charged
Choose differently when
During initial adoption, show costs without charging them for a year; charging from day one makes the cheapest option not meshing at all.
Why it holds up over time
A declared budget with a measured actual is the only cost control that survives changes in hardware prices, proxy efficiency and fleet size, because it compares ratios rather than absolute numbers. OpenCost reads Kubernetes resource data that any scheduler-level tool can provide.
LessonA shared platform's cost must be a number someone owns and every tenant can see, or it grows until it is noticed at the worst time.
Shown on views26 25 27

Every package used, in one table

Every open-source package named on the views, what it is, the job it does in this design, and what was considered instead.

PackageWhat it isWhat it does hereConsidered instead
Istio Service mesh, CNCF graduated Control plane (istiod), sidecar injection, mTLS, authorisation, traffic policy, multi-cluster, revision upgrades Linkerd, Cilium service mesh, Kuma
Envoy L4/L7 proxy, CNCF graduated Sidecar, east-west gateway and egress gateway data plane linkerd2-proxy
Kubernetes Gateway API Kubernetes routing API with a mesh profile HTTPRoute and GRPCRoute for weights, cohorts, mirroring and timeouts Istio VirtualService
SPIRE SPIFFE reference implementation, CNCF graduated Node and workload attestation, SVID issuance, 7-day in-cluster CA, issuance audit istiod CA, cert-manager with istio-csr
spire-controller-manager and SPIFFE CSI driver SPIRE's Kubernetes integrations Registration entries from ClusterSPIFFEID templates; Workload API socket into pods Hand-written entries; hostPath mounts
OpenBao Secrets and PKI server, Linux Foundation fork of Vault Per-cluster intermediate PKI mounts; signing key for bundles Vault (BSL), step-ca
OpenSSL with a PKCS#11 provider Cryptographic toolkit and HSM interface Offline root ceremony on an air-gapped workstation Vendor HSM tooling
Kubernetes Container orchestrator Workload platform, native sidecars, ValidatingAdmissionPolicy, audit log Nomad
Cilium eBPF CNI, CNCF graduated Pod networking, BGP route and VIP announcement, L3/L4 policy as second enforcement plane Calico with MetalLB
Argo CD GitOps reconciler, CNCF graduated Per-site delivery of mesh intent in cluster waves, with kind allowlists per project Flux
Argo Rollouts Progressive delivery controller Sole writer of canary weights; analysis on Thanos; automatic abort Flagger
Forgejo Self-hosted Git service Three intent repositories with signed commits and protected branches; mirrors per site GitLab CE, Gitea
Harbor OCI registry, CNCF graduated Mirrored mesh images and signed configuration bundles for one-step revert Zot
cosign and ORAS Artefact signing and OCI packaging tools Sign and package rendered bundles; verified at admission Notation
OPA Gatekeeper Policy admission controller for Kubernetes, part of OPA (CNCF graduated) Context-dependent admission rules; the same Rego as CI via conftest Kyverno
Open Policy Agent General policy engine, CNCF graduated Phase 3 external authorisation sidecar for opted-in services; CI policy checks Custom authz service
PostgreSQL with CloudNativePG Relational database and its Kubernetes operator SPIRE datastore per cluster; Forgejo database MySQL
Prometheus Metrics system, CNCF graduated Per-cluster scraping of proxy and control-plane metrics; recording rules VictoriaMetrics agent
Thanos Long-term storage and global query for Prometheus 13-month downsampled history on Ceph; site-local query for gates and analysis VictoriaMetrics, Grafana Mimir
OpenTelemetry Collector Vendor-neutral telemetry pipeline Node-local bounded buffering of access logs, spans and audit events Fluent Bit, Vector
ClickHouse Columnar analytical database Access logs (30 days) and issuance records (13 months); impact replay queries OpenSearch
Grafana Tempo Trace store on object storage Per-hop spans, 7 days Jaeger
Grafana and Alertmanager Dashboards and alert routing Per-service views, drift and cost dashboards; pages routed by failure origin Perses
OpenCost Kubernetes cost allocation Proxy and gateway cost attributed to teams against the mesh budget Spreadsheet from kube-state-metrics
Ceph RGW S3-compatible object storage Thanos and Tempo blocks; Object Lock bucket for the audit trail MinIO
chrony and node-problem-detector Time synchronisation daemon and node condition reporter Hold clock skew within ±60 s and taint nodes that drift beyond it ntpd, custom DaemonSet
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.
Solution Architecture v1.0 · Platform Networking Architecture · 2026-09 · 31 views · open source, fully on-premises31 views · HTML / SVG / draw.io