Evidence ledger
29 sources
Checked 14 Oct 2026
Evidence ledger
One row per claim in The proxy beside every process: when the sidecar's bill came due: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Dig: The proxy beside every process — what the per-request sidecar actually costs, who measured it, and why the industry built it, regretted it, and is now removing it. Research date: 2026-10-14. All rows checked 2026-10-14.
Retrieval limitation, stated up front. This session's egress policy allowed direct fetches
only to github.com and raw.githubusercontent.com. GitHub-hosted rows (the KEP text and its
git history, the two KubeCon slide decks mirrored on GitHub, the gRPC design doc, the MOSN
README) are first-hand: the files were downloaded and read in full. Every other row rests on
search-engine retrieval, which returns page excerpts and quoted fragments rather than the whole
document. Quoted fragments below are reproduced only where the retrieval returned them as
quotations; everything else is marked as a retrieved summary. The machine link check in
verify.mjs could not run against the network, so non-GitHub links are recorded as indexed and
unverified-by-HTTP rather than re-checked. Where a figure appears in only one excerpt, the page
says so.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Monzo (Oliver Beattie) | Anatomy of a Production Kubernetes Outage, KubeCon EU 2018 keynote slides | talk | 2018-05 | 2026-10-14 | https://raw.githubusercontent.com/sbueringer/kubecon-slides/master/slides/2018-kubecon-eu/Keynote%20Anatomy%20of%20a%20Production%20Kubernetes%20Outage%20-%20Oliver%20Beattie%2C%20Head%20of%20Engineering%2C%20Monzo%20Bank%20-%202018-04%20KubeCon%20outage%20keynote.pdf | Timeline: linkerd suspected 2 min in; restarts begin 6 min in; total outage at 1h03; incompatibility found and empty services deleted at 1h17; outage ends at 1h21 (slides pp. 36–73 of the PDF text) | Slide text, fetched first-hand: "⏰ 2 MINS INTO THE OUTAGE / Linkerd identified as unhealthy"; "⏰ ESCALATED TO TOTAL OUTAGE 1 HR 3 MINS INTO THE OUTAGE / Linkerd NullPointerException observed on start up"; "⏰ END OF OUTAGE 1 HR 21 MINS" |
| 2 | Monzo (Oliver Beattie) | Same keynote, root-cause slides | talk | 2018-05 | 2026-10-14 | https://raw.githubusercontent.com/sbueringer/kubecon-slides/master/slides/2018-kubecon-eu/Keynote%20Anatomy%20of%20a%20Production%20Kubernetes%20Outage%20-%20Oliver%20Beattie%2C%20Head%20of%20Engineering%2C%20Monzo%20Bank%20-%202018-04%20KubeCon%20outage%20keynote.pdf | Mechanism: Kubernetes 1.6 changed an empty endpoints list from [] to null, and linkerd failed on parse; three root causes named |
Slide text, first-hand: "endpoints": [] / "K8S < 1.6" vs "endpoints": null / "K8S 1.6+"; "ROOT CAUSES: Bug in gRPC client library affecting etcd / Incompatibility between Kubernetes + Linkerd / Human error" |
| 3 | Monzo (Oliver Beattie) | Major outage 27/10/2017, community post-mortem thread | postmortem | 2017-10-30 | 2026-10-14 | https://community.monzo.com/t/resolved-current-account-payments-may-fail-major-outage-27-10-2017/26296 | The published postmortem of record for the linkerd outage; restarting several hundred linkerd instances was the recovery decision that exposed the incompatibility | Retrieved summary: the postmortem records linkerd identified as suspect at 14:18 and a decision at 14:26 to restart several hundred linkerd instances; a newer linkerd tested in staging was rolled forward |
| 4 | InfoQ | Monzo Outage Post Mortem | blog | 2017-11 | 2026-10-14 | https://www.infoq.com/news/2017/11/Monzo-Outage-Post-Mortem | Independent contemporary account: the etcd cluster had been upgraded and expanded from three to nine nodes two weeks earlier; restarting linkerd made things worse | Retrieved summary: "Two weeks before the outage, the platform team had upgraded its etcd cluster... and expanded it from three nodes to nine"; "Restarting linkerd made things worse... a failure to parse empty services" |
| 5 | GitHub | GitHub availability report: August 2026 | postmortem | 2026-09 | 2026-10-14 | https://github.blog/news-insights/company-news/github-availability-report-august-2026/ | The Aug 17 2026 outage: an Istio sidecar hit its concurrency limit and did not scale because the autoscaling policy watched the host service rather than the sidecar; impact ~7h47m, web/API error rates ~20% at peak | Retrieved summary across the report and its coverage: sidecar "hit its concurrency limit and failed to autoscale, because a scaling policy was misconfigured to watch the host service rather than the sidecar's own limits"; remediation includes "correct autoscaling policies to cover sidecar capacity, audit Istio concurrency and scaling limits" |
| 6 | Lorin Hochstein (independent) | Tough days at GitHub, a continuing series | blog | 2026-08-18 | 2026-10-14 | https://surfingcomplexity.blog/2026/08/18/tough-days-at-github-a-continuing-series/ | Independent analysis of the same incident; notes the retry amplification and the gap between GitHub's status-page framing and the sidecar explanation | Retrieved summary: the analysis discusses the HAProxy exhaustion and retry storm; a commenter notes the blog post "blames Azure 'capacity failures', which isn't quite matching up with 'istio autoscaling broke'" |
| 7 | The Register | GitHub blames 8-hour outage on autoscaling fail and VS Code retry storm | blog | 2026-08-19 | 2026-10-14 | https://www.theregister.com/saas/2026/08/19/github-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm/5289547 | Blast radius corroboration: four HAProxy nodes exhausted connection limits; a VS Code retry bug raised Copilot Token Service traffic from a normal 7,000–9,000 rps to 70,000–100,000 rps | Retrieved summary: "four HAProxy load-balancer nodes exhausted their connection limits"; Copilot Token Service traffic "rose from a normal 7,000 to 9,000 requests per second up to 70,000 to 100,000" |
| 8 | Alpaca | Elevated Error Rates Across multiple APIs, postmortem | postmortem | 2026-01 | 2026-10-14 | https://status.alpaca.markets/incidents/8gnz4n33l574 | A weekend Istio rollout caused Monday connection instability; market-open traffic amplified it (incident recorded 2026-01-12 by a third-party tracker) | Retrieved quote from the status page: the investigation found "the Istio layer was not stable in establishing connections between services and components" |
| 9 | Meta (Saokar et al.) | ServiceRouter: Hyperscale and Minimal Cost Service Mesh at Meta, OSDI '23, pp. 969–985 | paper | 2023-07 | 2026-10-14 | https://www.usenix.org/conference/osdi23/presentation/saokar | At Meta, roughly 99% of RPC requests are routed by a library linked into the service executable; sidecar or remote proxies carry about 1% | Retrieved abstract text: "uses sidecar or remote proxies for only about 1% of RPC requests"; the mesh routes "tens of billions of requests per second across tens of thousands of services" (in production since 2012) |
| 10 | Meta (Saokar et al.) | ServiceRouter, same paper | paper | 2023-07 | 2026-10-14 | https://www.usenix.org/conference/osdi23/presentation/saokar | The stated reason for the library-first design is hardware cost at fleet scale | Retrieved abstract text: the embedded-library approach "significantly reduces the hardware costs of our hyperscale service mesh, saving hundreds of thousands of machines" |
| 11 | Meta (Saokar et al.) | ServiceRouter, same paper | paper | 2023-07 | 2026-10-14 | https://www.usenix.org/conference/osdi23/presentation/saokar | Sharded services account for 68% of RPC requests in Meta's fleet, a workload class most sidecar meshes do not route natively | Retrieved paper text: "sharded services, which account for 68% of RPC requests in our fleet" (a conference abstract says 92%; the paper figure is taken as authoritative and the discrepancy noted) |
| 12 | UW/Duke/ByteDance (Zhu et al.) | Dissecting Overheads of Service Mesh Sidecars, SoCC '23, pp. 142–157 | paper | 2023-10 | 2026-10-14 | https://foci.uw.edu/papers/socc23-meshinsight.pdf | Measured Istio/Envoy sidecar overhead on benchmark applications: request latency +27% to +269%, CPU +42% to +163% | Retrieved paper text: "request latency increases by 27-269% and CPU usage increases by 42-163%" across two benchmark apps |
| 13 | UW/Duke/ByteDance (Zhu et al.) | Same paper | paper | 2023-10 | 2026-10-14 | https://foci.uw.edu/papers/socc23-meshinsight.pdf | Where the cost lives: in HTTP/gRPC proxy mode, protocol parsing is 63–77% of total overhead; in TCP mode, inter-process communication dominates | Retrieved paper text: "When configured as an HTTP or gRPC proxy, protocol parsing alone represents 63-77% of the total overhead"; in TCP mode "most overhead stems from inter-process communication" |
| 14 | UW/Duke/ByteDance (Zhu et al.) | Same paper, fleet-trace analysis | paper | 2023-10 | 2026-10-14 | https://foci.uw.edu/papers/socc23-meshinsight.pdf | Applied to a large call-graph dataset, a quarter of call graphs would pay up to 100 ms added latency and 200 extra virtual cores | Retrieved paper text: "using Envoy increases latency by up to 100ms and consumes 200 more virtual CPU cores for a quarter of the call graphs" |
| 15 | Sharaf et al. | Performance Comparison of Service Mesh Frameworks: the mTLS Test Case (arXiv:2411.02267) | paper | 2024-11 | 2026-10-14 | https://arxiv.org/pdf/2411.02267 | Independent benchmark: ambient-mode Istio was the most memory-efficient mesh tested and had far lower latency growth than sidecar Istio at 3,200 RPS; Cilium was the most CPU-efficient | Retrieved summary of the technical report's findings (academic benchmark, single test harness) |
| 16 | Kubernetes SIG Node | KEP-753 Sidecar Containers, README and kep.yaml | adr | created 2018-05-14, updated 2025-01-23 | 2026-10-14 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers | The platform's own record that meshes ran on workarounds: service meshes could not guarantee start-before and stop-after ordering | KEP text, fetched first-hand: "as none of these are possible to guarantee, most service meshes (like Linkerd and Istio), use fragile and platform specific workarounds to have the basic functionality" |
| 17 | Kubernetes SIG Node | KEP-753 kep.yaml milestones | adr | 2025-01-23 | 2026-10-14 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/753-sidecar-containers/kep.yaml | Native sidecars: alpha v1.28, beta v1.29, stable v1.33; KEP created 2018-05-14 | kep.yaml, first-hand: creation-date: 2018-05-14; milestone: alpha: "v1.28" beta: "v1.29"; latest-milestone: "v1.33", stage: stable |
| 18 | Kubernetes contributors | kubernetes/enhancements git history for the sidecar KEP | source | 2019–2023 | 2026-10-14 | https://github.com/kubernetes/enhancements/commit/8c3cfd00db2523d0bc39e58e7c9a70a5c1b6cd49 | The 2019-approved design was demoted: marked implementable 2019-06-19 (commit d4277c2d), demoted to provisional 2020-06-24 and gated on a KEP that did not yet exist, then rewritten from scratch 2023-02-09 (commit 0dc6db75) | Commit diff, read first-hand from the cloned repo: "-status: implementable / +status: provisional"; added text: "On June 23 2020, during SIG-node meeting, it was decided that this KEP has a prerequisite on the (not yet submitted KEP) kubelet node graceful shutdown" |
| 19 | Kubernetes SIG Node | KEP-753 2023 rewrite, Alternatives section | source | 2023-02-09 | 2026-10-14 | https://github.com/kubernetes/enhancements/commit/0dc6db753668b2e7fc515d82963abbe0ba1a79f4 | The rejected designs are recorded: a pod.spec.SidecarContainers array, a primary-container marker, a Sidecar: true boolean, terminationFatalToPod, and depends-on semantics all lost to reusing init containers with restartPolicy: Always |
Commit diff, first-hand: removed TOC entries name each rejected alternative; new text records restart-backoff special-casing "which was proposed and rejected" |
| 20 | Kubernetes community | Issue #25908: Better support for sidecar containers in batch jobs | source | 2016-05 | 2026-10-14 | https://github.com/kubernetes/kubernetes/issues/25908 | The request that started the nine-year arc was filed in May 2016 and closed only after native sidecars shipped | Spotify's KubeCon slide (first-hand, row 21): "May 2016: Issue for 'Better support for sidecar containers in batch jobs' created"; Crunchy Data retrieval: "the popularity of the following issue on the Kubernetes issue tracker (which has now been closed!)" |
| 21 | Spotify (James Wen) | Kubernetes Jobs and the Sidecar Problem, KubeCon EU 2019 lightning talk slides | talk | 2019-05-20 | 2026-10-14 | https://raw.githubusercontent.com/mm-k8s-ug/kubecon-slides/master/slides/2019-kubecon-eu/Lightning%20Talk%20Kubernetes%20Jobs%20and%20the%20Sidecar%20Problem%20-%20James%20Wen%2C%20Spotify%20-%20Kubernetes%20Jobs%20and%20the%20Sidecar%20Problem%20Lightning%20Talk%20-%20James%20Wen%2C%20Spotify.pdf | Spotify ran three workarounds in production for the jobs-with-sidecars problem: shared-volume signalling, preStop termination calls, and a custom controller that watches for job completion and terminates the pod | Slide text, fetched first-hand: "Solution: Shared Volume Communication ... /shared-volume/main-terminated"; "Solution: Custom Controller ... terminate pod (when it sees job container is finished)" |
| 22 | Red Hat | Istio sidecar termination issue when running Jobs on OpenShift 4 | vendor | 2021 | 2026-10-14 | https://access.redhat.com/node/6280111 | The sanctioned workaround was the job calling the proxy's kill endpoint: POST to /quitquitquit on the Envoy admin port, with a sleep so the proxy is up before it is asked to die |
Retrieved summary: guidance uses "curl -sf -XPOST 127.0.0.1:15000/quitquitquit", notes the image needs curl, and adds a short sleep first |
| 23 | gRPC (Mark D. Roth) | gRFC A27: xDS-Based Global Load Balancing | adr | 2020-03-18 | 2026-10-14 | https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md | The proxyless path is a designed artefact, not a hack: gRPC clients speak Envoy's xDS control-plane protocol directly, status Implemented in C-core, Java and Go | Proposal text, fetched first-hand: "In order to converge with this industry trend, gRPC will be moving from its original grpclb protocol to the new xDS protocol"; header: "Status: Implemented" |
| 24 | Google Cloud | Traffic Director supports proxyless gRPC | vendor | 2021 | 2026-10-14 | https://cloud.google.com/blog/products/networking/traffic-director-supports-proxyless-grpc | Google's stated motive for proxyless: CPU cost of the per-pod proxy | Retrieved summary: Google's launch post says its own testing found proxyless gRPC "can save on networking-related CPU costs compared to sidecar proxies"; no numbers published |
| 25 | Netflix (Vroom, Mulcahy, Yuan, Gulewich) | Zero Configuration Service Mesh with On-Demand Cluster Discovery | blog | 2023-08-29 | 2026-10-14 | https://netflixtechblog.com/zero-configuration-service-mesh-with-on-demand-cluster-discovery-ac6483b52a51 | Netflix adopted Envoy sidecars in 2023 but refused the config tax: it built on-demand cluster discovery with the Envoy community so proxies fetch cluster config on first use instead of predefining it | Retrieved from the syndicated copy and InfoQ: "Proxies could now look up cluster information the first time they attempt to connect to it, rather than predefining all of the clusters in config"; trade-off acknowledged: "this adds latency to the first request" |
| 26 | InfoQ | Netflix zero-config service mesh coverage | blog | 2023-09 | 2026-10-14 | https://www.infoq.com/news/2023/09/zero-config-service-mesh-netflix | Netflix's alternatives considered: owner-defined clusters, config generated from the call graph, and pushing all clusters to every proxy were each rejected before on-demand fetch | Retrieved summary of the alternatives and their drawbacks as reported by InfoQ's coverage of the Netflix post |
| 27 | Istio project | Introducing Ambient Mesh | vendor | 2022-09-07 | 2026-10-14 | https://istio.io/latest/blog/2022/introducing-ambient-mesh/ | Istio's own design rationale for removing the sidecar: per-pod proxies must reserve for the worst case, injection restarts pods, and upgrades restart the fleet; ztunnel is a shared per-node resource | Retrieved summary of the announcement's stated sidecar drawbacks; the ambient split is L4 ztunnel per node plus optional L7 waypoint per service account |
| 28 | Istio project | Fast, Secure, and Simple: Istio's Ambient Mode Reaches General Availability in v1.24 | vendor | 2024-11-07 | 2026-10-14 | https://istio.io/latest/blog/2024/ambient-reaches-ga/ | Ambient reached GA in v1.24: ztunnel, waypoints and APIs marked Stable, two years after the September 2022 announcement; sidecar mode remains supported | Retrieved summary: ztunnel, waypoint proxies and APIs "now marked Stable"; the project "remains committed to continued sidecar support" |
| 29 | Istio project | Performance and Scalability (versioned docs) | vendor | 2019–2024 | 2026-10-14 | https://istio.io/latest/docs/ops/deployment/performance-and-scalability/ | Istio's own published tax per sidecar, by version: 1.2 added 8 ms to p90; 1.5.4 added 2.8 ms; 1.8.3 used 0.35 vCPU and 40 MB per 1,000 rps; 1.21 uses about 0.5 vCPU per 1,000 rps (benchmark: 1,000 services, 2,000 sidecars, 70,000 rps mesh-wide) | Retrieved doc text by version: "The Envoy proxy adds 8ms to the 90th percentile latency" (v1.2.5); "adds 2.8 ms" (v1.5.4); "0.35 vCPU and 40 MB memory per 1000 requests per second" (v1.8.3); "a proxy consumes about 0.5 vCPU per 1000 requests per second" (1.21-era docs) |
| 30 | Solo.io | What Istio Ambient Mesh means for your wallet | vendor | 2022 | 2026-10-14 | https://www.solo.io/blog/what-istio-ambient-mesh-means-for-your-wallet | The vendor's own test found no clear infrastructure saving once L7 waypoints are used: waypoints consumed about what sidecars did, and in the L4-only scenario ztunnel was the largest memory consumer | Retrieved summary: "Waypoint proxies used about the same resources as sidecars, and in the L4-only scenario ztunnel was the largest memory consumer"; defaults cited: ztunnel 200m CPU/512Mi per node, waypoint 100m/128Mi per replica, sidecar reservation 100m/128Mi |
| 31 | Solo.io | Istio ambient mode on EKS announcement | vendor | 2024-03 | 2026-10-14 | https://www.solo.io/press-releases/aws-enterprise-service-mesh | The marketing counter-claim, labelled as such: "cost savings of up to 80%" for existing Istio users moving to ambient | Retrieved press-release text: cites "cost savings of up to 80%" and quotes customer Trust Bank on lower compute cost |
| 32 | Red Hat | Introducing OpenShift Service Mesh 3.2: Istio's ambient mode | vendor | 2025 | 2026-10-14 | https://www.redhat.com/en/blog/introducing-openshift-service-mesh-32-istios-ambient-mode | A second vendor measurement of removing sidecars: memory reductions over 90% and CPU over 50% versus sidecar mode, larger when only ztunnel is used; test setup not published | Retrieved summary: "memory reductions of over 90% and CPU reductions of over 50% compared with sidecar mode"; the post does not give the test setup |
| 33 | Linkerd (William Morgan) | Why Linkerd doesn't use Envoy | adr | 2020-12-03 | 2026-10-14 | https://linkerd.io/2020/12/03/why-linkerd-doesnt-use-envoy/ | The recorded decision to build a purpose-built Rust micro-proxy instead of shipping Envoy beside every pod, because a general-purpose proxy is too heavy for N-per-cluster deployment | Retrieved quote: using Envoy "wouldn't allow them to build the lightest, simplest, and most secure Kubernetes service mesh"; Morgan's comparison: Envoy "172 KLOC" of C++ vs about "30 KLOC" for linkerd2-proxy |
| 34 | TechTarget | Linkerd service mesh production users will soon have to pay | blog | 2024-02 | 2026-10-14 | https://www.techtarget.com/searchitoperations/news/366570820/Linkerd-service-mesh-production-users-will-soon-have-to-pay | The economics of maintaining the lightweight data plane surfaced in 2024: companies over 50 employees would pay $2,000 per cluster per month for stable releases from May 21 2024; the CNCF opened a health check; pricing was later reworked to pod-based tiers | Retrieved text: "any company with more than 50 employees running Linkerd in production must pay Buoyant $2,000 per Kubernetes cluster per month to access stable releases"; follow-up coverage: "Pricing had been reworked into a three-tier scheme based on blocks of 100 Kubernetes pods" |
| 35 | Buoyant (William Morgan) | eBPF or Not, Sidecars are the Future of the Service Mesh (The New Stack) | blog | 2022 | 2026-10-14 | https://thenewstack.io/ebpf-or-not-sidecars-are-the-future-of-the-service-mesh/ | The dissent, on the record: per-host proxies trade isolation for density | Retrieved quotes: the sidecar-free pitch "replaces sidecar proxies with multitenant per-host proxies — a significant step backward for both security and operability"; "the assertion that eBPF needs to replace sidecars is a marketing construct, not an actual requirement" |
| 36 | Isovalent (Thomas Graf) / InfoWorld | Cilium 1.12 launches eBPF service mesh | blog | 2022-07 | 2026-10-14 | https://www.infoworld.com/article/3667493/cilium-launches-ebpf-powered-kubernetes-service-mesh.html | The opposite camp: Cilium 1.12 shipped a sidecar-less mesh using eBPF plus a per-node Envoy, supporting both models | Retrieved quote from Graf: "Enterprises want the ability to choose sidecars or sidecar-less"; Isovalent slides describe "eBPF + per-Node proxy - No sidecars" |
| 37 | Airbnb | Seamless Istio Upgrades at Scale | blog | 2025 | 2026-10-14 | https://airbnb.tech/uncategorized/seamless-istio-upgrades-at-scale/ | The operational bill of the sidecar at scale: Airbnb's mesh handles tens of millions of QPS at peak across tens of thousands of pods and thousands of VMs, has been upgraded 14 times, runs sidecar mode exclusively, and needed in-house tooling (Krispr, mxagent, mxrc) to upgrade without coordinating teams | Retrieved summary via InfoQ (2025-08): "tens of millions of queries per second at peak"; "completed Istio upgrades 14 times to date"; Airbnb "uses sidecar mode exclusively and does not yet run Ambient" |
| 38 | DoorDash | Staying in the zone: how DoorDash used a service mesh to manage data transfer, reducing hops and cloud spend | blog | 2024-01 | 2026-10-14 | https://careersatdoordash.com/blog/staying-in-the-zone-how-doordash-used-a-service-mesh-to-manage-data-transfer-reducing-hops-and-cloud-spend/ | The sidecar earning its keep: DoorDash fed AZ locality into Envoy via its in-house control plane so callers prefer same-zone callees, cutting cross-AZ transfer spend; no percentage published | Retrieved summary: DoorDash "modified its in-house custom service mesh control plane to provide Envoy with the AZ information for each node"; the published post gives no savings percentage |
| 39 | Ant Group | MOSN README | source | checked 2026-10-14 | 2026-10-14 | https://github.com/mosn/mosn | The sidecar model at its largest published non-Meta scale: Ant's Go sidecar proxy | README text, fetched first-hand: "verified by hundreds of thousands of production containers in 11.11 global shopping festival" |
| 40 | Bitnami | charts issue #5835 | source | 2021 | 2026-10-14 | https://github.com/bitnami/charts/issues/5835 | The jobs-never-complete failure reached ordinary users: a Kafka provisioning job hangs under injection, cross-referencing istio/istio#6324, #11659 and #15041 | Retrieved summary: the issue lists istio/istio#6324, #11659, #15041 as the upstream threads for a provisioning job that never finishes because of the sidecar |