For a decade the default answer to "how do we get uniform mTLS, retries and telemetry across a polyglot fleet" was to bolt a proxy onto every workload. This guide reconstructs what that proxy actually costs per request and per pod, from the teams that measured it, the outages it caused, and the three exits the industry is now taking. After reading it you can put a number on the tax for your own workload and defend where the data plane should live.
The problem, who has solved it in production, and the finding that reorganises how you should read everything else about service meshes.
State the problem without naming a product: several hundred services, written in several languages by teams that do not coordinate, all need encrypted transport, identity, retries, timeouts and request-level telemetry, and the platform team cannot get a library adopted into every one of them. The sidecar was the answer that required no application change: intercept the pod's traffic with iptables and route it through a proxy container injected next to the app. Istio, Linkerd and Ant Group's MOSN all ship this shape, and Ant's README reports it "verified by hundreds of thousands of production containers" during the 11.11 shopping festival, which is the largest sidecar deployment with a published scale figure.
The surprise in this corpus is who never bought in. Meta's ServiceRouter paper (OSDI '23) describes a mesh in production since 2012 that routes tens of billions of requests per second, and it reports that sidecar or remote proxies carry only about 1% of RPC requests; the other 99% are routed by a library linked into the service executable, a choice the paper justifies in one line: it "significantly reduces the hardware costs of our hyperscale service mesh, saving hundreds of thousands of machines". Google built the same exit for gRPC in 2020, teaching the client to speak Envoy's xDS control-plane protocol directly so no proxy is needed (gRFC A27, status Implemented). The organisations with the most requests concluded that a per-request toll multiplied by their traffic was a machine-count problem, and paid an engineering-coordination price instead. Everyone who could not coordinate their fleet paid the toll.
And the toll turned out to be large enough to reverse the trend. Istio announced a sidecar-free "ambient" mode in September 2022 and shipped it as GA in v1.24 in November 2024; Red Hat reports memory reductions over 90% when customers move off sidecars, a vendor figure with no published test setup; Cilium built its mesh around a per-node proxy from the start. Meanwhile the Kubernetes platform itself only made the sidecar a first-class concept in v1.33 (2025), nine years after users first asked. The pattern was deployed at bank scale for most of a decade on what its own enhancement proposal calls "fragile and platform specific workarounds".
Scope. This guide covers one decision and its consequences: where the service-to-service data plane runs, and what each placement costs in latency, compute, lifecycle coupling and operations. It does not cover whether you need a mesh at all, mesh control-plane scaling, ingress and API gateways, or eBPF as a general networking layer; those appear only where a source used them to decide placement.
The mechanism matters because every published failure in section 4 lives in one of these bolts, not in the proxy's routing logic.
The injected data plane has four load-bearing parts, and all four are invisible to the application. A mutating admission webhook rewrites the pod spec at creation time to add the proxy container and an init step. The init step programs iptables (or an eBPF equivalent) so every byte in and out of the pod detours through the proxy. The proxy holds mTLS identity, applies routing and retry policy, and emits telemetry. And a control plane pushes configuration to every proxy over xDS: routes, endpoints, certificates. Netflix's 2023 account of adopting Envoy sidecars is the cleanest description of the config problem this creates: a proxy nominally needs every upstream cluster predefined, which at Netflix meant thousands of clusters per proxy or a bespoke fix. They built the fix, on-demand cluster discovery, with the Envoy community, so a proxy fetches cluster config the first time it is needed, accepting added latency on the first request.
Kubernetes treated the proxy as an ordinary container until v1.28. The consequences are recorded in KEP-753 itself: a Job's pod never completes while the proxy runs, the proxy is often the first container OOM-killed because its requests are smallest, and start ordering cannot be guaranteed, so "most service meshes (like Linkerd and Istio), use fragile and platform specific workarounds to have the basic functionality". That sentence is the platform describing its own flagship pattern.
Source: KEP-753, fetched 2026-10-14
Each sidecar reserves CPU and memory sized for that pod's worst case, multiplied by every pod in the mesh. Istio's ambient announcement names this as a core reason to leave the model; the per-node ztunnel shares one reservation across the node's pods and keeps L7 parsing (the expensive part, 63–77% of overhead per MeshInsight) out of the default path entirely.
Sources: Istio, 2022; MeshInsight, 2023
Because the proxy lives in the pod spec, upgrading the mesh means restarting the fleet. Airbnb, which runs sidecar mode exclusively across tens of thousands of pods and thousands of VMs at tens of millions of requests per second, reports 14 completed Istio upgrades and built three in-house tools (Krispr, mxagent, mxrc) so upgrades do not require coordinating workload teams. That tooling bill is part of the sidecar's price.
Source: Airbnb, 2025, via InfoQ
The escape architectures remove bolts rather than tuning them. Ambient moves interception to a per-node L4 proxy and makes L7 an opt-in waypoint per service account; nothing is injected into pods, so upgrade and lifecycle coupling disappear. The library model (Meta, and Netflix's own pre-Envoy stack of Eureka and Ribbon) removes the extra process entirely and pays in coordination: Meta can do this because its fleet shares a build and deploy system, which is exactly the condition most enterprises fail. Proxyless gRPC keeps the central control plane but moves enforcement into the client, trading away everything Envoy can do that a gRPC channel cannot. None of these is free; they move the cost to a place a given organisation can or cannot afford.
Each fork with the recorded choice, the recorded rejection, and the condition that flips the answer.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Kubernetes sidecar API shape | Init containers with restartPolicy: Always |
A new pod.spec sidecar field, a primary-container marker, a boolean flag, depends-on semantics (all recorded as alternatives) |
Reusing init containers expresses start ordering without a new container class; the 2019 field-based design was demoted in June 2020 and never shipped | KEP rewrite, 2023 |
| Netflix: how proxies learn clusters | On-demand cluster discovery | Owner-defined clusters; config generated from the call graph; pushing all clusters to every proxy | Each rejected option either burdens service owners or bloats every proxy; on-demand fetch costs first-request latency instead | Netflix via InfoQ, 2023 |
| gRPC: where policy executes | xDS spoken natively by the client | Keeping grpclb plus a proxy on the path | Converge on the xDS standard and remove the hop; Google cites networking CPU cost savings in its own testing, without publishing numbers | gRFC A27, 2020 |
| Cilium: both or neither | eBPF plus a per-node Envoy, supporting sidecar and sidecar-less | Mandatory per-pod proxies | Graf: "Enterprises want the ability to choose sidecars or sidecar-less"; Morgan's counter is that per-host proxies are "a significant step backward for both security and operability" | InfoWorld, 2022; Morgan, 2022 |
Show the argument, not only the verdict: the Morgan–Graf exchange is the live disagreement in this field. Morgan's position is that the sidecar's per-pod isolation is a security boundary worth its cost and that per-node proxies reintroduce multi-tenancy inside the data plane. Graf's position is that enforcement belongs in the kernel and the node. Both are vendors defending their architecture; the measurement papers sit closer to Graf on cost and are silent on isolation, and no public incident yet adjudicates the security claim either way. That absence is a finding: the case against shared node proxies is argued entirely from first principles, not from postmortems.
Two failure classes account for everything published: the proxy as a new single point on every path, and the proxy as a component the platform cannot see.
null instead of [] and crashed with a NullPointerException on startup. A partial outage became total at 1h03./quitquitquit endpoint after a sleep. Ordinary users hit it through packaged charts (bitnami/charts #5835).restartPolicy: Always that starts before and terminates after the app. Alpha v1.28, beta v1.29, stable v1.33 (2025).The class that is missing matters too. No public postmortem in this corpus attributes an incident to the shared per-node proxy model (ambient's ztunnel, Cilium's per-node Envoy) despite two years of GA deployments. Read that with care: it is consistent with the model being safer, and equally consistent with its deployments being younger and smaller. The sidecar's failure record is partly a seniority effect.
Every figure with its context and date. The spread between vendor benchmarks and independent measurement is itself the lesson.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| p90 latency added per Envoy sidecar | 8 ms → 2.8 ms | Istio (vendor bench) | Istio 1.2.5 vs 1.5.4, same published benchmark series; big early-version effect | 2019–2020 | Istio docs |
| Proxy CPU per 1,000 req/s | 0.35 → ~0.5 vCPU | Istio (vendor bench) | 1.8.3 (with 40 MB memory) vs 1.21-era docs; bench is 1,000 services, 2,000 sidecars, 70k req/s mesh-wide | 2021–2024 | Istio docs |
| Latency overhead, measured independently | +27% to +269% | MeshInsight testbed | Istio/Envoy sidecars on two benchmark apps; client and server sidecars both on the path | 2023 | SoCC '23 |
| CPU overhead, measured independently | +42% to +163% | MeshInsight testbed | Same study; protocol parsing is 63–77% of overhead in HTTP mode | 2023 | SoCC '23 |
| Projected tax on real call graphs | up to 100 ms, 200 vCPU | MeshInsight on a trace set | For a quarter of analysed call graphs, added latency and added cores from full sidecar deployment | 2023 | SoCC '23 |
| Share of RPCs through any proxy | ~1% | Meta | The other 99% use the in-process library; mesh routes tens of billions of req/s | 2023 | OSDI '23 |
| Hardware saved by not using sidecars | hundreds of thousands of machines | Meta | The paper's own cost justification for the library-first mesh | 2023 | OSDI '23 |
| Memory saved leaving sidecar mode | >90% (claimed) | Red Hat customers | Vendor claim for ambient-style deployment, no test setup published; CPU claim >50% | 2025 | Red Hat |
| Cost saved moving to ambient | up to 80% (claimed) vs ~0% (measured) | Solo.io | The press-release claim and the same vendor's wallet analysis, where waypoints consumed about what sidecars did | 2022–2024 | Solo.io |
| Default reservations | sidecar 100m/128Mi; ztunnel 200m/512Mi per node; waypoint 100m/128Mi | Istio defaults via Solo.io | The arithmetic basis for any per-cluster estimate; multiply the sidecar line by pod count | 2022 | Solo.io |
| Sidecar mesh at peak | tens of millions req/s | Airbnb | Tens of thousands of pods, thousands of VMs, 14 completed upgrades, sidecar mode only | 2025 | Airbnb |
| Sidecar containers in production | hundreds of thousands | Ant Group | MOSN README's own scale claim for 11.11 festival traffic | checked 2026 | MOSN |
| GitHub outage impact | 7h47m, ~20% errors | GitHub | Peak web/API error rate; retry storm drove one service from 7–9k to 70–100k req/s | 2026-08 | GitHub |
| Monzo outage impact | 1h21m | Monzo | Total platform outage from 1h03; most payments still succeeded | 2017-10 | Keynote slides |
| Data-plane maintenance, priced | $2,000/cluster/month at launch | Buoyant/Linkerd | Stable-release access for companies over 50 employees, from May 2024; later pod-based tiers | 2024 | TechTarget |
Measured: the MeshInsight rows, Istio's own benchmark series (vendor-run but methodology published), and the incident figures. Claimed: Red Hat's 90%, Solo.io's 80%, Google's unnumbered CPU saving, and Meta's machine count (internal data nobody outside can audit). Derived: a cluster estimate you should do yourself, pods × the 100m/128Mi default row, which at 10,000 meshed pods reserves on the order of 1,000 vCPUs and 1.25 TB before a single request flows. Unknown, because nobody has published it: DoorDash's actual cross-AZ savings percentage, any fleet-wide dollar figure for sidecar spend outside Meta's machines claim, and every number in this table for your protocol mix, which MeshInsight shows can swing the answer by an order of magnitude.
Every source behind this page, graded. GitHub-hosted documents were fetched first-hand this session; the rest were retrieved through search excerpts, as the ledger records. Filter by kind.
The canonical sidecar-era incident: an etcd change starves linkerd of updates, a mass proxy restart hits a platform serialisation change, and the routing tier takes the bank down. Written by the engineers, in public, within days.
The slide deck, fetched first-hand from a GitHub mirror, carries the minute-level
timeline and the exact mechanism: "endpoints": [] under Kubernetes
pre-1.6 versus null from 1.6, which linkerd could not parse.
The sidecar failure class, nine years on: a proxy saturates at its concurrency limit while autoscaling watches the application's metrics, HAProxy exhausts, client retries amplify. Remediations name sidecar-aware autoscaling explicitly.
A brokerage rolls Istio into production over a weekend; Monday's market open finds the mesh "not stable in establishing connections between services and components". A small postmortem, but the only published account of mesh adoption itself as the incident.
The counterexample at the largest published scale: tens of billions of requests per second, in production since 2012, with ~1% of RPCs through any proxy. The library-first design is justified as saving hundreds of thousands of machines. Also documents native routing for sharded services, 68% of Meta's RPCs, which sidecar meshes do not route.
The measurement study: +27–269% latency and +42–163% CPU on benchmark apps, protocol parsing at 63–77% of overhead in HTTP mode, IPC dominant in TCP mode, and a predictive model showing a quarter of real call graphs would pay up to 100 ms and 200 vCPUs.
Independent benchmark across sidecar Istio, ambient Istio, Linkerd and Cilium: ambient the most memory-efficient, Cilium the most CPU-efficient, and sidecar Istio with the steepest latency growth at 3,200 requests per second.
The platform's own account of the gap, fetched first-hand: jobs that cannot complete, OOM-kill ordering that targets the proxy, unguaranteed start order, and the sentence that meshes ran on "fragile and platform specific workarounds". Created 2018-05-14; stable in v1.33.
Read from the cloned repository: marked implementable 2019-06-19; demoted to provisional 2020-06-24 after a SIG-node meeting gated it on a KEP that did not yet exist; rewritten from scratch 2023-02-09 with the 2019 API designs recorded as rejected alternatives.
The origin of the nine-year arc, dated by Spotify's slide deck and cross-referenced by downstream issues; closed only after native sidecars shipped.
Fetched first-hand: Spotify's three production workarounds for jobs with sidecars, drawn out slide by slide, ending with a call to action on the then-new KEP.
The lifecycle failure reaching ordinary users: a packaged Kafka chart's provisioning job hangs under injection, cross-referencing istio/istio #6324, #11659 and #15041.
The sanctioned workaround in a vendor knowledge base: the job's own container POSTs
to the proxy's /quitquitquit admin endpoint, after sleeping long enough
for the proxy to be ready to be killed.
Fetched first-hand. The design that makes the proxy optional for gRPC: the client speaks the mesh control-plane protocol itself. Status: Implemented in C-core, Java and Go.
Netflix moves from in-process libraries (Eureka, Ribbon) to Envoy sidecars for polyglot reasons, and the post records what that choice costs in configuration: three rejected designs, then an Envoy feature built upstream so proxies fetch clusters on first use.
The project arguing against its own default: per-pod proxies reserve for the worst case, injection restarts workloads, upgrades restart the fleet. Ambient splits L4 to a node ztunnel and L7 to optional waypoints.
Ztunnel, waypoints and the APIs marked Stable, two years after announcement, with a commitment to continued sidecar support.
The vendor's own benchmark series, useful precisely because it is versioned: p90 adder falling from 8 ms (1.2.5) to 2.8 ms (1.5.4), CPU per 1,000 req/s drifting from 0.35 (1.8.3) to about 0.5 vCPU (1.21 era) as default telemetry grew.
A vendor measuring its own product's savings and publishing the inconvenient result: waypoints consumed roughly what sidecars did, and in L4-only mode ztunnel was the largest memory consumer. The same company's press release claims up to 80% savings.
Claims memory reductions over 90% and CPU over 50% against sidecar mode, larger when only ztunnel is used. No test setup published; treat as a claim.
A public record of a data-plane decision: a purpose-built Rust proxy over Envoy, argued from size, memory safety and the N-per-cluster deployment model.
The maintenance economics surfacing: $2,000 per cluster per month for stable releases, community anger, a CNCF health check, and a later rework to pod-based pricing.
The strongest published argument for keeping the sidecar: per-host proxies are "a significant step backward for both security and operability", and the eBPF framing "a marketing construct". A vendor position, argued in mechanism.
The opposite camp ships: eBPF where possible, one Envoy per node where not, and both deployment models supported, with Graf's framing that enterprises want the choice.
The sidecar run well, at full disclosure of what that takes: 14 upgrades across tens of thousands of pods and thousands of VMs at tens of millions of requests per second, with three in-house tools so workload teams never participate in an upgrade.
The sidecar paying rent: DoorDash fed availability-zone locality into Envoy through its in-house control plane so callers prefer same-zone callees, attacking the cross-AZ transfer bill. No savings percentage is published.
README fetched first-hand: "verified by hundreds of thousands of production containers in 11.11 global shopping festival". The largest self-reported sidecar deployment outside the Envoy world, and a reminder the pattern is not only Istio's.
Google's managed control plane driving xDS-native gRPC clients, with the motive stated as saving networking-related CPU versus sidecars, in Google's own unpublished testing.
Independent reading of the GitHub incident, including the observation that the status-page framing (capacity) and the availability report's mechanism (sidecar autoscaling) do not quite line up.
Six rungs. The point is to produce your own column for the numbers table, because MeshInsight's central finding is that the tax is workload-specific.
Two small services on a kind or k3s cluster, calling each other directly. Drive them with a load generator (fortio or wrk2) at a fixed rate and record p50/p99 latency and per-pod CPU at that rate.
Done when: you have a table of latency and vCPU at 1,000 req/s with no mesh. Teaches: a tax is invisible without a pre-tax measurement, which is why so few teams can state theirs.
Install Istio in sidecar mode, enable injection on the namespace, rerun the same load. Compute your delta per 1,000 req/s and put it next to Istio's published 0.5 vCPU figure and MeshInsight's ranges.
Done when: you can state your own p99 and vCPU delta and explain any gap from the published numbers via protocol mix. Teaches: vendor benchmarks are a reference point, not your number.
Run a CronJob in the meshed namespace and watch the pod refuse to complete. Fix it
the 2019 way (the /quitquitquit call) and then the native way
(Kubernetes 1.29+, proxy as an init container with restartPolicy: Always).
Done when: the Job completes under both fixes and you can say why the native one is not a workaround. Teaches: the lifecycle coupling that consumed nine years of ecosystem effort, in one afternoon.
Set the proxy's concurrency and CPU limit low, configure an HPA on the app container's CPU, and raise load until errors climb while the HPA sits still, which is GitHub's August 2026 incident in miniature. Then wire a sidecar saturation signal into scaling or alerting.
Done when: you can produce the failure and then make it impossible. Teaches: the platform scales what it measures, and injection changed what needs measuring.
Remove injection, enable ambient, measure L4-only, then attach a waypoint and measure again. You now hold the sidecar/ambient-L4/ambient-L7 comparison that Solo.io and Red Hat publish conflicting claims about, for your workload.
Done when: a four-row table: baseline, sidecar, ambient L4, ambient with waypoint. Teaches: whether the 90% memory claim or the no-saving measurement describes you.
Convert one gRPC client-server pair to xDS-native gRPC against the same control plane and measure the hop you removed. Note what you lost: mTLS story, L7 policy, non-gRPC traffic.
Done when: the pair runs with no proxy on the path and you can list the features you gave up. Teaches: the Meta/Google trade in miniature: toll-free traffic, paid for in generality.
The queries that found this material, grouped by what they surface. The vocabulary terms (ztunnel, waypoint, quitquitquit, proxyless, KEP-753) are the keys; each one was learned from a source and then searched.
"service mesh" sidecar "we removed" OR "we moved off" productionistio ambient "in production" sidecar-less migration"proxyless" gRPC xDS production latencyistio OR linkerd outage postmortem "root cause" -tutorial"availability report" sidecar autoscaling site:github.blogstatus.<vendor>.com incident istio rolloutsidecar overhead measured vCPU "per 1000 requests"MeshInsight OR "dissecting overheads" service meshambient mesh "wallet" OR "cost" waypoint ztunnel measuredKEP-753 sidecar containers history provisional implementableServiceRouter OSDI Meta sidecar "1%" library"kubecon-slides" filetype:pdf sidecar site:github.comGitHub-hosted documents were fetched in full this session; all other links were located and excerpted through search retrieval and are recorded as indexed rather than re-fetched, as the ledger (sources.md) states.