An API's latency histogram is bimodal. Most requests finish in about 40 ms and roughly 0.8% land between 240 ms and 1.3 s. The pattern is independent of endpoint and instance and payload size, server-side handler time is flat at 12 ms, CPU and queue depth are normal, and no error is logged. Where is the time going, and what do you change?

The first three things I would look at, and why in that order

1. The shape of the histogram, not its average. Two discrete clusters rather than a smooth tail means something is waiting on a fixed-duration timer, not doing variable work. Queueing, contention and garbage collection produce continuous distributions; timers produce steps.
2. Retransmission counters on both ends - segments retransmitted, and specifically SYN retransmits. These are the cheapest confirmation available and almost nobody charts them.
3. Whether the slow requests correlate with a path, not an instance: source and destination availability zone, and which member of an equal-cost multi-path bundle the flow hashed onto.

The diagnosis

Those two clusters are the two timers that govern recovery from packet loss.

• Linux clamps the retransmission timeout to a minimum of about 200 ms. A lost data segment that cannot be recovered by fast retransmit - which needs duplicate acknowledgements, so it needs following packets - waits for that timer. 200 ms plus a 40 ms baseline is the 240 ms cluster.
• A lost connection-opening SYN has no duplicate acknowledgements available at all, and its initial retransmission timeout is about 1 second. That is the 1.0 to 1.3 s cluster.

So the request is not slow; it is losing a packet at the worst possible moment - the last packet of a response, or the first packet of a connection. Payload-size independence is the clue that it is not bandwidth, and it is also what misleads people into excluding the network. A 0.1% loss rate produces roughly this frequency once you count every packet in every short exchange.

The misleading signal

Server-side handler time at a flat 12 ms. It is accurate, and it measures from the first byte of the request being parsed to the last byte being written to the socket - so it excludes both of the intervals where the time is going. Application timers do not see the network, which is why the network is the last thing anyone suspects.

The fix

1. Find and remove the loss. It is commonly one degraded member of a link bundle, one saturated uplink at a particular hour, or a middlebox dropping under its own connection-table pressure. Flow logs plus per-path probes locate it; a single bad optic is a cable swap.
2. Reduce exposure to the SYN case by reusing connections. Keep-alive on clients and pooling at every hop removes most connection establishments, so most of the 1 s cluster disappears even while loss persists. This is the fastest available mitigation.
3. Enable the modern recovery mechanisms - tail loss probe and RACK are on by default on current kernels and specifically target the lost-last-packet case. Confirm rather than assume.
4. Lower the minimum retransmission timeout only on paths you control end to end, inside one data centre where round trips are sub-millisecond. On internet paths a 200 ms floor exists to avoid spurious retransmits, and lowering it causes the problem it appears to solve.

The alert that would have caught it earlier

Retransmission rate and SYN-retransmit count per source-destination pair, charted next to p99. Average latency moved from 40 ms to about 46 ms when this started, which is invisible. Alert on the percentage of requests above a fixed threshold, not on the mean, because a bimodal distribution has no meaningful mean.

Common weak answers

• "Profile the application harder." Handler time is accurate and complete for what it measures, so more profiling produces more exoneration of the application and no progress.
• "Scale out." Capacity added to a path that is waiting rather than working, and spreading flows over more paths can include the degraded one more often.
• "Lower the retransmission timeout." Correct only inside a data centre you control end to end. On an internet path the 200 ms floor exists to prevent spurious retransmits, and lowering it produces more loss and a worse version of this problem.
• "It is the client's network." Possibly true, and unprovable without the retransmit counters - which is the work being avoided.