Lyft built Envoy and open-sourced it in 2016 so that retries timeouts load balancing and telemetry stopped being per-language library concerns. What did that buy what did it cost and what happens to a fleet of 2000 sidecars when the management server serving their xDS configuration is unreachable for 40 minutes during an autoscaling event?

The situation they were in

A polyglot estate migrating off a monolith, where every language had its own HTTP client, its own retry defaults, its own circuit-breaking behaviour and its own metric names. Two consequences, both documented as the motivation for the project: nobody could answer "which service is failing" consistently, because the measurements were not comparable, and changing a networking policy meant a library release in each language followed by a rebuild of every service that used it.

Envoy, created at Lyft and open-sourced in 2016, moves that work out of the process into a proxy beside it, configured over the xDS APIs by a management server. It is now a CNCF graduated project and the data plane underneath most service meshes.

What the sidecar bought

• One implementation of the hard parts, in one language, for every service: retries with budgets, timeouts, outlier detection, zone-aware load balancing.
• Comparable telemetry by construction. The same statistics with the same names for every hop, regardless of what the application is written in. This is the benefit that is hardest to replicate with libraries and the one teams underestimate.
• Policy changes that ship without a rebuild. A retry policy change is a configuration push, not an N-language library rollout - the difference between a change that reaches production in minutes and one that takes a quarter.

What happens when the control plane goes away

Per Envoy's own documentation, a proxy that loses its management-server stream latches on to its previous configuration and keeps retrying the connection in the background. So for 40 minutes:

• Existing traffic keeps flowing. Routes, clusters and endpoints already known continue to work. The data plane is deliberately not dependent on the control plane for steady state.
• Nothing new exists. The pods the autoscaler just started are never added to any endpoint list, so the new capacity receives no traffic while the old pods saturate - the failure looks like an autoscaler that is not working.
• Removals stop too. Terminated endpoints may stay in the last-known configuration, so callers keep sending requests to addresses that are gone until their own health checking ejects them.
• Certificate rotation stalls. If secrets are delivered dynamically, expiry arrives hours later as widespread TLS failures with no obvious connection to the morning's outage.
• Deploys appear to succeed and change nothing. A canary weight shift or a route change is accepted by the control plane and never reaches the proxies.

What you watch: each proxy's connected_state statistic, alerting on sustained disconnection measured in minutes, plus the age of the configuration each proxy is running. The proxies keep serving the configuration they already hold, which is what makes the silence survivable and also what makes it silent.

What it costs

A second process on every pod - order of tens to low hundreds of megabytes resident depending on configuration size, plus CPU that becomes material at high request rates - and a control plane that is now a tier-zero service with its own on-call, its own staged rollouts and its own blast radius. The mesh moves coupling rather than removing it: out of the build, into the runtime.

Where copying it would be a mistake

Ten services in one language. A shared client library delivers most of the policy and telemetry consistency at a fraction of the operating cost, and the rollout problem a mesh solves is not yet your problem. The mesh earns its cost when library rollout is the binding constraint - many languages, many teams, or a policy that must change faster than services can be rebuilt.

The second mistake is adopting the data plane without budgeting for the control plane. Envoy is the part that is given to you; the management server, its configuration pipeline, its staged rollout and its monitoring are the part you have to build and operate, and that is where the ongoing cost lives.

Common weak answers

• "The mesh removes networking concerns from services." It relocates them. The service no longer implements retries, and somebody now owns a fleet of proxies and the configuration that drives them.
• "If the control plane goes down the mesh goes down." The opposite of what the design does, and the reason the real failure is dangerous: steady state survives, so the outage presents as an autoscaler that is not scaling and certificates that expire later.
• "Run the control plane highly available and the problem goes away." Necessary and not sufficient. The property that saves you is the proxies' ability to run on stale configuration, and the thing to prove is that you can detect stale configuration before it matters.