A gRPC client fleet calls a backend behind a layer-4 load balancer. After scaling the backend from 10 to 40 pods, the original 10 carry almost all the traffic and the new pods sit idle. Separately, calls have started failing with UNAVAILABLE after quiet periods. Diagnose both.

The first three things to look at, in order

1. Requests per second per pod, never the aggregate. The distribution is the entire symptom and the average hides it.
2. Connection count and connection age per pod. Long-lived connections concentrated on the original pods is the confirmation.
3. Whether anything between client and server has an idle timeout — cloud NAT, the load balancer itself, a firewall.

Diagnosis one: the traffic will not move

gRPC runs over HTTP/2, which multiplexes many concurrent streams onto one long-lived TCP connection. A layer-4 balancer balances connections, not requests. The client connects once, lands on a pod, and every RPC for the next several hours rides that connection.

Scaling out therefore adds pods that no existing connection points at. New capacity receives traffic only when a new connection is made, which means only when a client instance restarts. This is the exact opposite of HTTP/1.1 with a connection per request, where an L4 balancer approximates request balancing by accident — which is why the problem surprises teams who have never had to think about it.

The fixes, in the order you should want them:

1. Client-side load balancing over individual endpoints. Resolve pod addresses directly (a headless service, or xDS from a control plane) and let the gRPC client distribute RPCs across subchannels. This is the intended model and the only one that gives true per-request balance.
2. A layer-7 proxy that understands HTTP/2 and balances individual streams.
3. MAX_CONNECTION_AGE on the server as a blunt mitigation: the server sends GOAWAY after a bounded lifetime — 300 seconds to 1800 seconds is a common setting — clients reconnect, and the fleet re-spreads. Add jitter or every client reconnects together. It costs a periodic reconnection storm and a small latency bump on the affected RPCs, which is why you prefer the first two options and reach for this one on the afternoon you need relief.

Diagnosis two: UNAVAILABLE after quiet periods

A middlebox is silently dropping idle TCP flows. Cloud NAT gateways, load balancers and firewalls all expire idle connections in production, commonly somewhere between 30 seconds and 600 seconds depending on the product, and they do it without notifying either end. The client's connection object still looks healthy; the next RPC writes into a flow that no longer exists and fails when its deadline expires.

The fix is HTTP/2 keepalive pings at an interval below the middlebox timeout, with the option that permits pings while no stream is active, because an idle connection is precisely the case. The server must be configured to allow that ping rate: gRPC servers enforce a minimum interval and respond to over-eager clients with GOAWAY and ENHANCE_YOUR_CALM, which turns a fix into a new outage if the two sides are configured independently.

The misleading signal

Fleet averages look healthy. CPU and memory across 40 pods are fine because 30 of them are idle, and the aggregate error rate is low for the same reason. The 10 loaded pods may be at their limit, and every dashboard that aggregates is actively concealing it. Distrust an average taken over a fleet you have not confirmed is evenly loaded.

When this is the wrong answer, and when not to build a control plane

If the client fleet is large and restarts frequently, connection churn spreads traffic well enough on its own and an xDS control plane is a large amount of machinery for a problem you do not have. And inside a service mesh the sidecar is already doing L7 balancing, so this failure cannot occur — the diagnostic question there is whether the traffic is actually going through the sidecar, which is a different investigation.