Retransmission Timeout Floor

A request path with a 40 ms baseline shows a bimodal histogram: most requests at 40 ms, and about 0.8% in two clusters, one near 240 ms and one near 1.1 s. Server-side handler time is flat. CPU, queue depth and payload size are all unremarkable, and no error is logged anywhere.

Discrete clusters mean a timer. Work produces continuous distributions: queueing, lock contention and garbage collection all smear. A constant means something waited a fixed interval, and in a request path there are two such constants worth memorising.

Linux clamps the retransmission timeout to a minimum of roughly 200 ms, so a lost data segment that cannot be recovered by fast retransmit - which needs duplicate acknowledgements, and therefore needs following packets - waits for that floor. A lost connection-opening SYN has no acknowledgements available at all, and its initial retransmission timeout is about 1 second.

Why it matters

It is the most commonly misattributed source of tail latency. The clusters sit far from the median, so they dominate p99 while barely moving the mean - in the example above, the average went from 40 ms to about 46 ms, which no alert would catch.

It also survives every application-level investigation. Handler timing is accurate and excludes both intervals, because it measures from request parse to socket write: the lost last packet of a response and the lost first packet of a connection both happen outside it. So profiling, tracing spans and flame graphs all exonerate the application, correctly, and the search continues in the wrong place.

The frequency is higher than intuition suggests. At 0.1% per-packet loss, every short exchange of a dozen packets has roughly a 1% chance of losing one, and the subset lost at a recovery-unfriendly moment lands in these clusters.

Implementation patterns

• Read the histogram shape first, not the percentile. Two bumps is a different diagnosis from a long smooth tail, and the bump positions name the cause.
• Chart retransmissions and SYN-retransmissions per source-destination pair. These are cheap counters, available on every host, and almost never collected.
• Correlate with path rather than instance - zone pair, and which member of an equal-cost multi-path bundle the flow hashed onto. Low-level loss is often one degraded member.
• Reuse connections aggressively. Keep-alive and pooling at every hop remove most connection establishments, which removes most of the 1-second cluster even while the loss continues. This is the fastest available mitigation and usually the cheapest.
• Confirm modern recovery is enabled - tail loss probe and RACK target precisely the lost-last-packet case and are default on current kernels, but "default" is worth verifying rather than assuming.
• Alert on the share of requests above a fixed threshold rather than on a mean, because a bimodal distribution has no meaningful mean.
• Lower the floor only on paths you control end to end, inside a data centre with sub-millisecond round trips.

Industry example

The 200 ms floor and the roughly 1 second initial timeout are long-standing kernel defaults, with the initial value standardised at 1 second in RFC 6298 (2011) after earlier specifications used 3 seconds. Published latency investigations from large operators repeatedly arrive at the same place: a tail that looks like application jitter resolves into timer-shaped clusters caused by small amounts of loss, and the fix is either removing the loss or removing the connection establishment, not tuning the service.

Failure scenarios

• Weeks spent profiling the application because handler time is flat and therefore trusted as complete.
• Scaling out in response, which adds capacity to a path that is waiting rather than working, and can worsen loss by spreading flows over more paths including the bad one.
• Lowering the floor on internet paths, which causes spurious retransmits, more loss and a worse version of the original problem.
• Averaging the histogram away. A dashboard showing mean latency hides the entire phenomenon.
• Blaming the client. The clusters appear in client-side measurement and not in server-side measurement, which makes "it must be their network" an attractive and often wrong conclusion.

Trade-offs

Connection reuse is almost free and is the right first move, but it trades memory and state-table entries at every hop for fewer handshakes, and long-lived connections pin flows to paths - including a degraded one - until they are recycled.

Chasing the loss itself is the correct structural fix and is often outside your control, sitting in a transit provider or a partner's network. The pragmatic sequence is to remove the exposure first, then pursue the cause with evidence.

When not to use it

This is the wrong explanation when the slow requests are not clustered at characteristic intervals. A smooth tail that scales with payload size points at bandwidth or at the application; a tail that correlates with one instance points at that instance.

It is also the wrong lens for QUIC and HTTP/3 traffic, which have their own loss-recovery timers and different defaults, and for paths where round trips are already hundreds of milliseconds, where the floor is below the measured round trip and therefore not a floor at all.

Interview question

Q: A service's p99 is 1.2 s while its p50 is 40 ms, the histogram has two tight clusters, and server-side timing is flat at 12 ms. Walk me through your diagnosis and your first three actions.

What a strong answer covers: reading the clusters as timers rather than work, and naming the roughly 200 ms data floor and roughly 1 s SYN timeout · why server-side timing cannot see either interval · checking retransmission and SYN-retransmission counters and correlating by path rather than by instance · connection reuse as the fastest mitigation and why it removes the larger cluster first · pursuing the loss to a specific path member or saturated uplink · and the alerting change: percentage over a fixed threshold, with retransmits charted next to p99.

Quick check

Quiz: Why do two tight latency clusters at about 240 ms and 1 s rule out application contention? Contention produces continuous distributions; discrete clusters mean fixed-duration waits, and those two durations are TCP's minimum retransmission timeout and its initial SYN timeout.

Flashcard: What is the fastest mitigation for a latency cliff caused by packet loss, before the loss itself is found? - Connection reuse: pooling and keep-alive remove most handshakes, so the roughly 1-second lost-SYN cluster largely disappears while the loss is still there.