Ephemeral Port Exhaustion

A TCP connection is identified by a four-tuple: source address, source port, destination address, destination port. When one host talks to one destination endpoint, three of the four are fixed, so the only degree of freedom is the source port — and the supply is finite, commonly about 28,000 on default Linux settings (the range 32768 to 60999).

Ports are not returned at close. The side that closes actively holds the tuple in TIME_WAIT for twice the maximum segment lifetime, 60 seconds on Linux, so that delayed segments from the old connection cannot be mistaken for part of a new one. That is correct behaviour and it sets the arithmetic.

At 2,000 new connections per second to one destination, steady-state TIME_WAIT occupancy is 2,000 × 60 = 120,000 tuples against roughly 28,000 available. Exhaustion arrives in about 14 seconds.

Why it matters

The error points at the wrong machine. connect() fails with EADDRNOTAVAIL, the application reports connection failures to a remote service, and the investigation goes to the remote service, which is perfectly healthy. Teams have lost days to this.

It also appears suddenly at a threshold rather than degrading, because the port supply is consumed and then gone. A service that has been fine for months crosses the rate at which churn exceeds recycling and fails within seconds of crossing it.

Implementation patterns

• Reuse connections. Keep-alive with a correctly sized pool eliminates the entire class, and it also removes the handshake latency and the TLS CPU that arrive with the same defect. This is almost always a client-library configuration problem, not a kernel tuning problem.
• Do the arithmetic per destination, not per host: connections-per-second × 60 seconds against the available range, for each destination endpoint separately.
• Enable tcp_tw_reuse for outbound connections, which lets the kernel reuse a TIME_WAIT tuple for a new outgoing connection when it is safe to. This is the supported knob; tcp_tw_recycle was broken behind NAT and has been removed from Linux.
• Widen the ephemeral range as a stopgap, which buys a linear factor and does not change the shape of the problem.
• Add destination entropy — more destination addresses or ports — so the four-tuple has more room, which is what a load balancer with multiple VIPs gives you.
• Alert on TIME_WAIT socket count and on EADDRNOTAVAIL, because both exist well before the failure.

Industry example

The failure is a standing feature of operating high-throughput HTTP clients, and it is the reason every mature HTTP client library ships connection pooling on by default and every cloud provider documents NAT gateway port allocation limits — a managed NAT multiplexes many instances through a small set of addresses, which makes the same four-tuple arithmetic much tighter than it is on a single host. Teams typically meet it the first time a service is scaled up in production behind NAT.

Failure scenarios

• Sporadic connection failures attributed to the remote service, for hours.
• Failure at a sharp threshold after months of stability, following a modest traffic increase.
• NAT gateway port allocation limits hit collectively by a fleet, so no single instance looks close to its own limit.
• A retry storm making it worse, because each retry is another connection and another 60 seconds of TIME_WAIT.
• Kernel tuning applied first, producing a widened range and the same bug at a slightly higher rate.

Trade-offs

Connection pooling costs idle connections held open — memory on both ends, file descriptors, and the possibility that a middlebox drops an idle flow without telling either side, which is a different failure requiring keepalives. tcp_tw_reuse trades a small theoretical risk from delayed duplicate segments for a large practical gain. Widening the ephemeral range takes ports from services that bind to them, so it is not free either, merely cheap.

When not to use it

This analysis does not apply when connections are spread over many destinations. Two thousand connections per second across 50 destination endpoints is 40 per second each, or 2,400 tuples in TIME_WAIT, comfortably within range — so a fan-out workload can churn connections for years without meeting this. Reaching for pooling or sysctls there solves nothing, and the diagnostic that separates the two cases is simply counting connections per destination, which takes one command and is skipped remarkably often.

Interview question

Q: A service starts failing to connect to one dependency roughly fifteen seconds after each deployment, with EADDRNOTAVAIL. The dependency is healthy throughout. Walk me through the diagnosis and the fix.

What a strong answer covers: the four-tuple and why only the source port varies for a single destination; the ephemeral range size and the 60-second TIME_WAIT hold; the arithmetic giving time-to-exhaustion from the connection rate; that the error is local and the dependency is a red herring; the fix order — connection reuse first, then tcp_tw_reuse, then range widening, then destination entropy; and the related idle-connection failure that keepalives address.

Quick check

Quiz: Why can a host open only about 28,000 simultaneous connections to one destination? The connection is identified by a four-tuple; three parts are fixed, so the count is bounded by the ephemeral source-port range, and closed connections hold their tuple in TIME_WAIT for 60 seconds.

Flashcard: Sporadic EADDRNOTAVAIL to one healthy dependency. What is happening and what do you fix first? Source-port exhaustion from connection churn plus 60-second TIME_WAIT. Fix connection reuse in the client library first; kernel tuning only buys a linear factor.