Accept Queue

A service reports healthy: p99 inside budget, CPU at 30%, error rate flat. Clients report 1% of connections timing out. Both are telling the truth, because the failing connections never reached the application. They completed their handshake, sat in a kernel queue, and were dropped there.

A listening socket has two queues. The SYN queue holds half-open connections mid-handshake; the accept queue holds connections the kernel has fully established and that are waiting for the process to call accept(). Its depth is the smaller of the backlog the application asked for and the system limit (net.core.somaxconn). When it is full, Linux by default drops the final handshake packet, so the client waits for a retransmission - the first retry typically about a second later - and the service records nothing at all.

Why it matters

It is the first place saturation appears and the last place anyone looks, because it sits between the two sets of metrics people trust. The client sees a timeout; the server sees no request. The symptom - a bimodal latency distribution with a cluster around one second and multi-second connection times at low CPU - is the signature, and it is routinely misdiagnosed as a network fault.

It also explains why the accept loop is a concurrency limit in its own right. If a process accepts connections on one thread and that thread also does work, accept throughput falls under load, the queue fills, and the service rejects connections while its workers are idle.

Implementation patterns

• Watch the counters. netstat -s reports listen queue overflows and SYN cookies sent; those counters rising is proof, not inference. They are per-host and free to scrape.
• Raise both numbers together. Setting somaxconn to 4096 does nothing if the application passes a backlog of 128 to listen(), which many frameworks default to.
• Accept on a dedicated path. Separate accepting from request handling, or use multiple listeners with SO_REUSEPORT so the kernel spreads connections across processes and no single accept loop is the ceiling.
• Prefer a short queue with explicit shedding to a long one: a deep accept queue is latency the client cannot see and cannot cancel, which is the bufferbloat argument applied to connection setup.
• Keep connections alive. The cheapest fix is usually to stop establishing so many: HTTP keep-alive, pooled upstream connections and HTTP/2 or gRPC multiplexing remove most of the arrivals.

Industry example

The mechanism is why connection-oriented front doors are tuned before application code in high-arrival-rate systems. Published tuning guidance for reverse proxies and load balancers consistently pairs a raised somaxconn with a raised application backlog and SO_REUSEPORT across worker processes, and the standard diagnostic for "intermittent connection timeouts with healthy servers" in production runbooks is the listen-overflow counter. The same queue is what makes a cold start expensive: a process that is slow to reach its accept loop silently accumulates a full queue, then drops everything.

Failure scenarios

• Mass reconnection after a deploy or a network blip. Every client arrives at once; the queue fills in milliseconds; retransmission backoff turns a one-second event into a thundering herd thirty seconds long.
• A blocked accept loop. One slow operation on the accepting thread - a DNS lookup, a lock, a log write to a full disk - stops accepting entirely while the service reports itself healthy.
• Health checks succeed because they are already-open connections, so the load balancer keeps sending traffic to an instance that can no longer accept new ones.
• TLS on the accept path. With the handshake performed on the accepting thread, CPU spent on cryptography directly reduces accept rate.
• Container limits. A low somaxconn inherited from a base image caps a service that was sized for ten times the arrival rate.

Trade-offs

A deeper accept queue buys tolerance of bursts and pays in hidden latency: a connection may sit for seconds while the client has already given up, so the service eventually does work nobody is waiting for. A shallower queue fails faster and more honestly, at the cost of refusing connections that a brief burst would have absorbed. The sound position is a modest queue plus a real admission policy - bound the queue, measure overflow, and shed with a response the client can act on rather than with a dropped packet.

When not to use it

This is not a knob to turn preemptively. Raising the backlog on a service whose problem is slow request handling moves the queue from the kernel to the application and makes diagnosis harder, and it is the wrong answer when connection arrivals are low and each request is expensive. If arrival rate is modest and overflow counters are zero, leave the defaults alone: the capacity problem is somewhere else, and a deep accept queue is latency you have chosen not to see.

Interview question

Q: Clients report 1% of requests timing out at about one second. Your service shows no errors, p99 of 40 ms and CPU at 30%. What do you check, and in what order?

What a strong answer covers: recognising the one-second cluster as a TCP retransmission rather than application latency; checking listen-overflow and SYN-cookie counters before anything else; the two-queue model and the min(backlog, somaxconn) rule; why server-side metrics are blind to it; the fixes in order (fix the backlog pair, take work off the accept loop, SO_REUSEPORT, keep-alive to cut arrivals); and the observation that health checks on existing connections can keep traffic flowing to an instance that cannot accept new work.

Quick check

Quiz: Why does accept-queue overflow produce client timeouts of roughly one second with no server-side error? The kernel drops the handshake's final packet, so the client waits for its first retransmission and the request is never seen by the application.

Flashcard: What two values set the accept queue depth? The backlog the application passes to listen() and the system's somaxconn - the effective depth is the smaller of the two.