A gRPC client reports UNAVAILABLE errors in bursts every few minutes. The errors cluster on connections that have been idle. The load balancer's idle timeout is 350 seconds and the client's keepalive is 600 seconds with a 20-second timeout. What is happening and what are the correct settings?

The trigger

The balancer closes an idle connection at 350 seconds and the client does not find out. A middlebox that drops or silently closes a flow leaves the client holding a socket it believes is open: no FIN was processed, nothing in the client's stack knows, and the connection is half-open. The next request written to it fails - immediately if a reset comes back, or after a timeout if the packets simply go nowhere.

Because the client's keepalive fires only at 600 seconds, there is a 250-second window in which the connection is dead and the client thinks it is alive. Idle connections in the pool accumulate in that state, which produces exactly the observed pattern: bursts of failures on the connections that have been quietest.

Why it propagated

Connection pools hand out connections without checking liveness, so one dead connection serves several callers in quick succession. And a retry on the same pool can draw another dead connection, turning one failure into a short run of them. The errors are UNAVAILABLE rather than a timeout, which is the clue that the transport failed rather than the server.

Why detection lagged

Nothing on the server side sees these failures at all. The request never arrived, so server error rates, latency and logs are clean, and the dashboards everyone trusts show a healthy service. The only places the truth appears are client-side error counters and the balancer's connection-close metrics, which are rarely graphed together.

The structural fix

• Client keepalive interval must be shorter than the shortest idle timeout on the path - the load balancer, any NAT device, and the server. With a 350-second balancer timeout, a keepalive of 60 to 120 seconds with a 20-second timeout keeps the connection demonstrably alive and detects death within one interval. The rule is a ratio, not a number: keepalive at roughly a third of the smallest idle timeout in the path.
• Enable keepalive while idle (permit_without_stream in gRPC terms). A keepalive that only runs when streams are active does nothing for the idle connections that are the problem.
• Set the server's permitted minimum keepalive interval consistently, or the server will send GOAWAY to clients it considers too chatty - the opposite failure, and a confusing one.
• Use max_connection_age on the server so connections are recycled deliberately with a GOAWAY, which both refreshes load-balancing decisions and makes closure a graceful event the client handles rather than a surprise.
• Retry on connect-time and transport failures only. These are safe to retry because the request never reached the server, and distinguishing them from application failures is what makes the retry policy both safe and effective.

The general lesson

The cost of the fix is small and real: a keepalive PING every 60 s on 10000 idle connections is 10000 round trips a minute of pure overhead, plus wakeups on devices that would rather sleep, which is why the HTTP/2 specification (RFC 7540, 2015) leaves the interval to the implementation and why servers enforce a minimum. Prefer the longest interval that is still comfortably below the smallest idle timeout in the path, rather than the shortest interval you can get away with.

Every long-lived connection crosses devices with their own idle timeouts, and the shortest one governs. The inventory worth writing down for any persistent-connection protocol - gRPC, WebSockets, database pools, message brokers - is the idle timeout of every hop, with the application's keepalive set below the minimum. The failure is otherwise invisible from the server, intermittent by nature, and routinely misattributed to the network.

When not to tune keepalive

If the client makes a request every few seconds, the connection is never idle and keepalive is irrelevant - the bursts of UNAVAILABLE have another cause, most often a balancer recycling connections without a graceful GOAWAY. And for a client that talks to a service a handful of times an hour, the simpler answer is to stop pooling: open a connection per call and accept the handshake cost, which removes the half-open problem entirely for a few milliseconds per request.