Clients report that roughly 1% of requests time out after about one second. The service shows no errors, p99 of 40 ms and CPU at 30%. Both measurements are honest. Where is the problem?

The first three things to look at, in order

1. The exact timeout value. About one second, with no server-side record, is the fingerprint of a TCP retransmission rather than of application latency. The client sent a packet, got nothing, and waited for its first retry.
2. Listen-queue overflow counters on the hosts. netstat -s reports listen-queue overflows and SYN cookies sent. A rising overflow counter is proof rather than inference, and it takes seconds to check.
3. The effective accept-queue depth, which is the smaller of the backlog the application passed to listen() and the system's somaxconn.

The diagnosis

The failing connections never reached the application. A listening socket has two kernel queues: the SYN queue for half-open connections and the accept queue for fully established ones waiting for the process to call accept(). When the accept queue is full, Linux by default drops the final handshake packet. The client, having completed its side, waits for a retransmission - typically about a second - and the server records nothing at all, because no request was ever read.

That is why both measurements are honest. p99 of 40 ms is the p99 of requests the application saw, and the failures are not in that population.

The misleading signal is CPU at 30%. It argues against saturation, and the saturated resource is not CPU: it is either the depth of the accept queue or the rate at which the process drains it. A single-threaded accept loop that also does work accepts more slowly under load, so the queue fills while the workers are comparatively idle.

The fix, in order

1. Raise both numbers together. Setting somaxconn to 4096 achieves nothing if the framework passes a backlog of 128, which many defaults do.
2. Take work off the accept path. Accept on a dedicated thread, or run multiple listeners with SO_REUSEPORT so the kernel spreads new connections across processes and no single loop is the ceiling. TLS handshakes on the accept thread are a frequent culprit.
3. Reduce arrivals rather than absorbing them. Keep-alive, pooled upstream connections, and HTTP/2 or gRPC multiplexing remove most of the connection establishment, which is the cheapest fix available and usually the right one.
4. Prefer a modest queue plus explicit shedding to a deep queue: a deep accept queue is latency the client cannot see and cannot cancel.

The alert that would have caught it earlier

Listen-queue overflows per host, and the ratio of accepted connections to attempted ones. Both are free to collect and neither appears on a standard application dashboard. Worth adding alongside them: a check that does not reuse an existing connection, because a health check on an already-open connection succeeds happily while the instance cannot accept new work - which is how a load balancer keeps sending traffic to a host that is refusing it.

When this is the wrong answer

If overflow counters are zero, the accept queue is exonerated and the one-second signature points at the next layers: ARP or neighbour-table exhaustion, a conntrack table at its ceiling on a NAT device in the path, an ephemeral port shortage at the client, or packet loss on one path. Raising the backlog on a service whose real problem is slow request handling is actively harmful - it moves the queue from the kernel into the application and makes the delay invisible to both sides. Check the counters first; they are the thing that distinguishes this fault from the half-dozen that look identical from the client.