A fashion marketplace's average latency is excellent but p99 is unacceptable for a small, commercially important customer segment. How should tracing, segmentation, queue analysis and dependency timing guide the investigation?

Why the average is misleading

An average is dominated by the common case and the tail is a different population. A system where 99% of requests take 50ms and 1% take 4 seconds has an excellent average and a terrible experience for one in a hundred requests — and because a page makes many requests, a 1% per-request tail affects a much larger percentage of page loads.

That composition effect is the reason tail latency matters more than it appears: with twenty requests per page, a 1% slow rate means roughly one in five page loads contains a slow request.

The investigation sequence

1. Segment before diagnosing. Slice p99 by customer tier, region, device, app version, account age and catalogue size. The tail is very often not random — it is one segment being consistently slow, which makes it a specific problem rather than a statistical one. A customer with ten thousand items in a wishlist is not experiencing variance; they are experiencing an unindexed query.
2. Compare client-observed with server-observed latency. A gap means time is spent in queueing, connection acquisition, DNS or the network — none of which appears in server-side timing, and all of which are common causes.
3. Look at the distribution shape. A long smooth tail suggests contention and queueing; a bimodal distribution suggests two code paths, usually a cache hit and a cache miss.
4. Trace the slow requests specifically, using tail-based sampling. A 1% head sample keeps 1% of the slow requests, which is almost none of them.
5. Check queue time separately from service time, since a fast handler behind a saturated pool produces a slow request with no slow component.

The usual causes, in rough order of frequency

Queueing at a saturated resource · cache misses on the long tail of data · garbage collection pauses · a dependency's own tail amplified by fan-out, where a request calling ten services each with a 1% slow rate is slow about 10% of the time · connection pool exhaustion · data-dependent work, where a large account does more work than a small one.

The structural fix that is often correct

Isolate the segment. If large accounts are consistently slow, the answer is frequently not to optimise the shared path but to give that segment different treatment: a dedicated pool, a precomputed view, a different query strategy. Optimising the average path to serve the tail makes the common case worse for everyone and usually still fails the tail.