A client library opens a new HTTPS connection for every request. One host sends 2,000 requests per second to a single destination endpoint. What breaks first, and in what order does everything else follow?

What breaks first, and when

Ephemeral source ports, in about fourteen seconds.

A TCP connection is identified by the four-tuple of source address, source port, destination address and destination port, so the port is the only degree of freedom you have here because the other three are fixed by the deployment. With one source IP and one destination endpoint, only the source port varies, so the number of tuples you can hold at once is bounded by the ephemeral port range — roughly 28,000 on common Linux defaults (32768 to 60999).

Ports are not returned immediately. The side that closes actively holds the tuple in TIME_WAIT for twice the maximum segment lifetime, 60 seconds on Linux. At 2,000 new connections per second, steady-state TIME_WAIT occupancy is 2,000 × 60 = 120,000 tuples against about 28,000 available. Exhaustion arrives in roughly 28,000 ÷ 2,000 ≈ 14 seconds.

What the user and the operator see

connect() begins failing with EADDRNOTAVAIL, surfacing in the application as sporadic connection errors. They look exactly like the remote being unavailable, and the remote is perfectly healthy — which is why this gets misdiagnosed for hours, with the investigation pointed at the wrong host entirely.

What happens before exhaustion

• TLS handshake CPU. A full handshake performs asymmetric cryptography; at 2,000 per second this is a large continuous CPU cost that appears as busy processors with no application work to account for it.
• Latency. Every request now pays a TCP handshake (one round trip) plus a TLS handshake (one round trip with TLS 1.3, two before it). A call with 1 ms of work to a service 5 ms away takes 15 to 25 ms, so the service looks slow while doing nothing wrong.

What stops it

1. Connection reuse. Prefer keep-alive with a correctly sized pool, which removes the entire class of problem and is the only fix that also removes the handshake latency and the CPU in production. This is almost always a client-library configuration defect rather than a kernel tuning problem, and reaching for sysctls first is how teams end up with a tuned kernel and the same bug.
2. If churn is genuinely unavoidable: widen the ephemeral range; enable tcp_tw_reuse, which is safe for outbound connections (unlike the long-removed tcp_tw_recycle); or add destination addresses so the four-tuple has more entropy.
3. TLS session resumption removes most of the handshake cost and none of the port cost, so it treats the symptom you noticed second.

What would have to be true for it to self-heal

Nothing. TIME_WAIT is a fixed-duration timer, and the only variables are your arrival rate and your port range — both of which are on your side of the wire. The condition persists exactly as long as the churn does.

When this is not the problem

When connections go to many distinct destinations. The limit is per four-tuple, so 2,000 connections per second spread over 50 destination endpoints puts each one at 40 per second, which is 2,400 tuples in TIME_WAIT and comfortably inside the range. Always do this arithmetic per destination, not per host.