A gRPC service sits behind a layer 4 load balancer. One backend receives most of the traffic and adding instances does not help. Why?

What is being tested

Understanding that connection-level and request-level balancing are different, and that HTTP/2's main efficiency feature defeats the former.

The mechanism

A layer 4 balancer distributes connections. With HTTP/1.1 that approximates distributing requests, because connections are relatively short-lived and numerous.

gRPC uses HTTP/2, which multiplexes many concurrent requests over a single long-lived connection. A client opens one connection and sends thousands of requests through it. The balancer made one decision — at connection time — and every subsequent request follows it.

So each client is pinned to one backend for the life of its connection. With a small number of high-volume clients, load is distributed by client rather than by request, which is arbitrary and usually badly skewed.

Adding instances does not help, because existing connections do not move. New instances receive traffic only when a new connection is established, which may be rarely or never.

The fixes

1. Layer 7 / request-level balancing. A proxy that understands HTTP/2 can distribute individual streams across backends. This is the direct fix and what most gRPC-aware ingress does.

2. Client-side balancing. The client resolves all backend addresses and maintains connections to several, distributing requests itself. This is the standard approach in large internal estates and gives the client information a central balancer lacks — which backends have been slow for its requests, and which are in its own availability zone.

3. A service mesh sidecar, which is client-side balancing implemented outside the application so it can be upgraded without redeploying every caller.

4. Connection lifetime limits — force clients to re-resolve and reconnect periodically. A blunt mitigation that rebalances over time; useful as a stopgap and not a solution.

The related trap

The same mechanism affects scale-in. Removing an instance drops all its multiplexed streams at once, which is far more disruptive than removing an HTTP/1.1 backend. Graceful shutdown must send GOAWAY, let clients establish new connections, and only then close.

Why the other options are wrong

A memory leak produces degradation over time, not immediate skew from deployment. Health check misconfiguration would remove backends entirely rather than skew traffic to one. gRPC does not require sticky sessions — the stickiness here is an accident of connection-level balancing, not a design requirement.

The broader point

This is a good example of a general phenomenon: an efficiency feature at one layer breaking an assumption at another. HTTP/2's connection reuse is exactly what makes it efficient, and it is exactly what invalidates connection-count-based balancing. Whenever a protocol changes the relationship between connections and requests, every component that counts one as a proxy for the other needs re-examination.