Connection Pinning

HTTP/1.1 with a connection per request made layer-4 load balancing look like request load balancing. It was never that; it only appeared to work because connections were short and numerous.

HTTP/2 — and therefore gRPC — multiplexes many concurrent streams onto one long-lived TCP connection. An L4 balancer places that connection once, and every RPC for the next several hours rides it to the same backend.

The consequence is counter-intuitive and appears at exactly the wrong moment: scaling the backend out does nothing. New pods exist, pass their health checks and receive no traffic, because no existing connection points at them. Capacity arrives only as clients reconnect, which for a stable client fleet may be never.

Why it matters

It breaks the assumption that adding instances adds capacity, which is the assumption every autoscaler and every incident runbook is built on. During a load event the fleet scales, the dashboards show plenty of headroom in aggregate, and the original pods remain saturated.

It also makes averages actively misleading. With 10 loaded pods and 30 idle ones, mean CPU, mean latency and aggregate error rate all look healthy while a third of the fleet is at its limit.

Implementation patterns

• Client-side load balancing over individual endpoints. Resolve backend addresses directly — a headless service, or endpoints from an xDS control plane — and let the client spread RPCs across subchannels. This is the intended model for gRPC and the only one that gives true per-request distribution.
• A layer-7 proxy that balances streams, such as a sidecar or an L7 load balancer with gRPC support, when client-side balancing is impractical.
• MAX_CONNECTION_AGE with jitter on the server as a blunt mitigation: GOAWAY after a bounded lifetime — 300 to 1800 seconds is common — forces reconnection and re-spreads the fleet. Without jitter every client reconnects simultaneously.
• Keepalive pings below the shortest middlebox idle timeout, with the server configured to permit that rate; gRPC servers enforce a minimum interval and answer over-eager clients with GOAWAY and ENHANCE_YOUR_CALM.
• Alert on the distribution, not the mean: per-pod RPS spread, or max/median ratio, so imbalance is visible before a load event.

Industry example

The problem is well documented in the Kubernetes and gRPC communities, where the default Service abstraction balances connections at layer 4 and the standard remedies are a headless service with client-side balancing or a mesh sidecar doing layer 7. It is the same mechanism that motivated Envoy's original design at Lyft: moving balancing into a process that understands the application protocol, so that decisions are made per request rather than per connection.

Failure scenarios

• Scaling with no effect, during the incident where scaling was the plan.
• Deploy-time thundering herd when every client reconnects at once after a rolling restart, concentrating on whichever pods came up first.
• Silent capacity waste: paying for 40 pods and serving from 10, for months, with healthy-looking dashboards.
• UNAVAILABLE after idle periods, when a NAT gateway or firewall drops the idle flow and neither end is told.
• Keepalive misconfiguration, where a client pings more often than the server permits and is disconnected for it, turning a fix into a new outage.

Trade-offs

Client-side balancing means every client needs endpoint discovery and the logic that goes with it — more moving parts in every application, and a control plane to run if you use xDS. A mesh sidecar moves that complexity into infrastructure at the cost of a proxy hop, extra latency of a millisecond or so, and resource overhead per pod. MAX_CONNECTION_AGE costs nothing to adopt and buys only partial relief, since traffic re-spreads periodically rather than continuously.

When not to use it

If the client fleet is large and restarts often, natural connection churn spreads traffic well enough, and building an xDS control plane solves a problem the deployment cadence is already solving. If you already run a service mesh, the sidecar does layer-7 balancing and this failure cannot occur — the useful question there is whether the traffic genuinely traverses the sidecar, which is a different investigation with a different answer.

Interview question

Q: You scale a gRPC backend from 10 to 40 pods under load and throughput does not improve. Aggregate CPU looks fine. Walk me through the diagnosis and the fixes in priority order.

What a strong answer covers: looking at per-pod RPS rather than the aggregate, and at connection age and count; the HTTP/2 multiplexing mechanism that makes an L4 balancer distribute connections rather than requests; why fleet averages conceal it; the fixes ranked — client-side balancing over resolved endpoints, then an L7 proxy, then MAX_CONNECTION_AGE with jitter as immediate relief; and the related idle-timeout failure with keepalive configured on both sides.

Quick check

Quiz: Why does adding gRPC backends behind an L4 load balancer not add capacity? Because HTTP/2 multiplexes all requests onto one long-lived connection, and the balancer distributes connections, so new pods receive traffic only when clients reconnect.

Flashcard: A third of your gRPC pods are idle and the rest are saturated. What do you change first? Move to client-side load balancing over individually resolved endpoints, or an L7 proxy that balances streams; use MAX_CONNECTION_AGE with jitter as immediate relief, not as the fix.