Architecture One-Pager

Service Mesh Platform · Solution Architecture v1.0 · Platform Networking Architecture · 2026-09 · 31 views · open source, fully on-premises

A mesh that is compiled ahead of time and enforced locally: every proxy forwards on three things it already holds, so the control plane can fail without taking a single request with it.

Two thousand services call each other across twenty clusters in three data centres. Today each team decides for itself whether to encrypt, how to authenticate a caller, how long to wait and how often to retry, and the answers are inconsistent in exactly the places where an incident starts. A mesh fixes that by moving those decisions into a proxy beside every workload. It also puts a new component on the path of every request the organisation serves, 40,000 copies of it, and that is the harder problem. A mesh that encrypts everything but goes down with its control plane, or costs 20% of the fleet's CPU, is a regression presented as a platform.

Intent is declared in three Git repositories (routing, authorisation, identity), checked by policy before merge and again at admission, and applied to each cluster by a site-local Argo CD in waves. In each cluster, istiod compiles that intent into per-workload Envoy configuration, scoped to what the workload is declared to call, and streams it as deltas. SPIRE issues each workload a 24-hour X.509 identity derived from what the kubelet reports about it, chained through a 7-day in-cluster CA and a name-constrained per-cluster intermediate in OpenBao to a root that lives on an offline HSM. Envoy sidecars terminate and originate mTLS, evaluate authorisation in-process, route by weight and header, and shed load before it queues. Clusters in different sites meet through east-west gateways that route on SNI and hold no identity. Cilium provides pod networking, BGP-announced gateway addresses and a second, independent enforcement plane. Telemetry leaves every proxy through a bounded node-local buffer into Prometheus and Thanos, Tempo and ClickHouse, tagged with whether a failure came from the application, the mesh or the network.

What it is, and what it is not

• Configuration compiled ahead of the request — not a control plane, CA or policy service asked anything while a request is being forwarded.
• Identity attested by the platform — not a name the workload chooses, an IP address, or a namespace label anyone with kubectl can set.
• Authorisation decided at the destination proxy — not a synchronous call to a central policy server on every request, except where a service opts in and declares the latency.
• One writer per kind of intent — not routes, policy and weights edited by whoever has cluster-admin, racing each other during an incident.
• A proxy per workload, with a node-level L4 option later — not a shared per-node L7 proxy multiplexing many identities in one process from day one.
• A freeze when the control plane is down — not an outage, and also not a promise that a new deny lands while it is down.

The decisions that are the architecture

1. The control plane compiles; it is never consulted (ADR-01) — Every proxy holds its last acknowledged configuration, a trust bundle and an unexpired certificate, and forwards on those alone. A total control-plane outage freezes intent; it does not stop traffic.
2. Istio on Envoy, with the standard APIs preferred (ADR-02) — The most widely deployed open mesh, a CNCF graduated project, with the L7 features the requirement needs today and a per-node L4 mode for later. Routing is written in the Kubernetes Gateway API wherever it has the field.
3. SPIRE attests; the workload never names itself (ADR-07) — Identity comes from a node attested by projected token and a process the kubelet vouches for. Every issuance is recorded with that evidence for 13 months.
4. 24-hour certificates are the containment and the outage budget (ADR-08) — Renewal at 12 hours gives the fleet 12 hours of issuer outage with no degradation, and a stolen key expires within a day. The two numbers were chosen together.
5. A cluster can only mint its own identities (ADR-09) — Each cluster intermediate is name-constrained to its trust domain, lives in OpenBao, and is signed once a year at an offline HSM ceremony. Proxies trust roots only.
6. Authorisation in the proxy, dry-run first, impact-gated (ADR-11) — Policies compile into the destination proxy and cost microseconds. A change is shadowed against live traffic and refused at merge if it would deny more than 0.5% of a service's current calls.
7. One owner of weights, and revert is the fastest path (ADR-15) — Argo Rollouts is the only writer of canary weights and reads back the share the mesh applied. Reverting a mesh change pins a signed bundle that was already rendered, ahead of Git.
8. Every retry is paid for from a budget (ADR-17) — Timeouts are mandatory, retries are limited to idempotent methods and 20% of active requests per destination, and deadlines travel with the call so nested retries cannot multiply.

Why it holds up over time

A mesh is adopted for a decade and its components are replaced every two or three years. The design is arranged so that the replacements are local: each seam between components is an open standard or a documented contract, and the decisions that encode organisational judgement (who may change what, what a request depends on, how long a credential lives) are kept out of any one product's configuration language.

• The seams are standards, not products. SPIFFE for identity, xDS for proxy configuration, the Kubernetes Gateway API for routes, OTLP for telemetry. Linkerd, Cilium and a future Envoy-based mesh all speak at least three of them, so replacing Istio is a migration of objects rather than a rewrite of the platform's contract with 250 teams.
• The dependency direction is the architecture. Nothing on the request path depends on the control plane, the CA or the configuration store. That rule survives a change of proxy, of mesh, even of scheduler. It is what keeps a replacement of any component low-risk, because the component being replaced was never on the path.
• The coupled numbers are written down together. Certificate lifetime, renewal point, the outage the fleet survives and the clock-skew budget are one decision, recorded in one place. A future team shortening certificates to an hour will see, in the same record, that they have also turned a 30-minute CA outage into an outage.
• Intent outlives the tool that applies it. Routes, policies and dependencies live in Git as reviewed, signed history, with the sha stamped on every applied object and reported back by every proxy. Argo CD, istiod or both can be replaced and the audit trail, the drift join and the separation of duties carry on unchanged.
• Upgrades are a routine, not a project. Revision-based upgrades run two mesh versions side by side and move namespaces in waves, every quarter. A platform that upgrades four times a year never falls out of upstream support and never faces a three-version jump that turns into a programme.
• The cost model scales with the fleet, not with hope. The idle floor of 40,000 proxies is measured and attributed to teams monthly. When the per-node L4 data plane matures, the saving is a measured decision per namespace, and the architecture already has a place for it.
• The root is rehearsed, not remembered. A ten-year root is replaced by people who were not there when it was created. Rotating a staging mesh's root every year keeps the ceremony, the overlap arithmetic and the runbook current long before production needs them.
• Failure is assumed, and drilled. Each of the ten failure classes has a detection signal, a containment and a quarterly game-day. Assumptions that are exercised stay true as the system changes; assumptions that are only documented quietly stop being true.

Non-functional targets

The requirement's targets, how this design meets each one, and the view where the mechanism is drawn. Where a figure is a planning estimate, it is the first number the proof phase replaces.

Scope

In scope

• Workload identity, issuance, rotation, revocation and the trust hierarchy from root to SVID
• mTLS for east-west traffic, permissive and strict modes, and the plaintext report
• Destination-side authorisation with default deny, dry-run and an impact gate
• Subset routing, 1% weights, header cohorts, mirroring, locality routing and one-step revert
• Timeouts, budgeted retries, deadline propagation, outlier ejection and load shedding
• Sidecar lifecycle, injection policy, configuration scoping, delta distribution and fleet upgrades
• Multi-cluster service federation through east-west gateways, and controlled egress
• Per-hop metrics, traces and access logs, failure attribution, drift and cost attribution

Explicitly out of scope

• The north-south edge: public ingress, WAF, DDoS protection and end-user authentication
• Workload scheduling, cluster provisioning and node lifecycle
• Application-level authorisation that needs user entitlements or session state (the external-authz hook is the seam for it)
• Traffic to databases and brokers that clients reach without passing through a sidecar
• A self-service portal; changes are reviewed in Git
• Any managed cloud service; every component runs on hardware the organisation owns

Prove it before production depends on it

Eight weeks, one site, two clusters, 400 synthetic services with a real dependency graph and three real services migrated. Each item below replaces an assumption in this record with a measurement or retires a product behaviour from the list above.

1. Measure idle CPU and memory per sidecar at 400 and at 2,000 synthetic services, scoped and unscoped
2. Measure added p50 and p99 per hop at 500, 1,000 and 2,000 rps per proxy, with authz and stats on
3. Sign 2,000 SVIDs a second from one SPIRE server set on CloudNativePG, and restart 2,000 pods at once
4. Confirm Envoy refuses an SVID signed outside its intermediate's name constraint
5. Apply the retry-budget and expected-timeout EnvoyFilter, then upgrade Istio one minor version with it in place
6. Run a dry-run policy at scale and compare its would-deny log with the impact replay of the same change
7. Take istiod, SPIRE and OpenBao down for 13 hours under load and record what changes at hours 0, 12 and 13
8. Fail a canary on purpose and time detection, revert and convergence end to end

• Pass: mesh-attributed failures stay under 0.01% throughout the 13-hour outage, and the first failures after 12 hours are renewal alerts, not requests.
• Pass: rollback converges on 95% of caller proxies within 60 seconds of the analysis breach.
• Fail and rethink: idle CPU per scoped sidecar above 20 millicores, which moves the per-node L4 layer from Phase 3 into Phase 2.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 32 records across 7 areas, each with the alternatives that lost and what the choice costs.