A service's p99.8 is 4 seconds against a p50 of 30 ms. Roughly one request in 500 is slow and it is not reproducible outside production. CPU is 35% and no instance looks unusual. Which instrumentation locates the cause first?
Show the full answer Hide the answer
The deciding property
The slow population is 0.2% of requests, and every aggregate instrument averages it away. That single fact ranks the options. The first question is not "which code is hot" but "which phase of the request consumed the 4 seconds", and phases are what traces measure. Only once the span is known does a profiler have a useful target.
Why tracing wins here
A trace is per-request, so a slow request produces its own complete record, and 0.2% of a million requests a day is still 2,000 exemplars. Tail-based sampling keeps the slow ones specifically, which is what makes 0.2% affordable to retain.
The decisive property of this latency profile is that a 4-second p99.8 against a 30 ms p50 is almost never CPU. That ratio is the signature of waiting: a connection-pool acquisition, a lock, a retry with a one-second backoff, a DNS lookup that missed cache, a garbage-collection pause, or a downstream that is itself tailing. All of those appear in a trace as a gap or a long span, and none of them appear in a CPU profile at all, because a blocked thread burns no cycles. CPU at 35% is the clue that says so.
Why the other options fail
- An on-demand profiler attached when the alert fires. Two problems. It samples whatever is running during the window, so the 0.2% contributes 0.2% of samples and vanishes into noise. And by the time a human attaches it, the slow request is finished. This is the right tool for a sustained CPU regression, not an intermittent tail.
- Always-on continuous profiling. Genuinely useful, and the honest reason it is not first here is attribution: unless the profiler carries per-request labels (goroutine labels or their equivalent) and you can filter to the slow cohort, it reports fleet averages dominated by the fast 99.8%. With request-scoped labels it becomes the right second step, after a trace names the span.
- Kernel syscall tracing on one instance for an hour. Low overhead and very high resolution, and it answers "which syscalls are slow on this host". If the cause is a pool, a lock or a downstream, it produces a wall of correct detail about the wrong layer. Reach for it once a trace points at something the userspace view cannot explain, such as page-cache misses or a stalled write.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| p50 and p99 both rise together and CPU is 85% | continuous profiling | the cost is in code that is running, not waiting |
| the slow span is already known to be one function | an on-demand profile of that path | the population problem is solved; now you need line-level cost |
| the gap sits inside a single span with no sub-spans | syscall or kernel tracing | the time is below the application's visibility |
| the service has no tracing at all | add spans to the top five call sites first | three days of work beats three weeks of guessing |
Common weak answers
- "Add more metrics." A histogram tells you the tail exists. It cannot tell you which phase of which request produced it, and this problem is already past that point.
- "Reproduce it in staging." Not reproducible outside production is given. The useful response is to instrument production, not to argue with the premise.