A request takes 3 seconds. Every individual service reports healthy latency. How does tracing resolve this and what must have been instrumented?

What is being tested

Whether you understand that per-service aggregates cannot reveal a distributed latency problem, and what tracing needs in order to be useful when it matters.

Why every service can look healthy

Several mechanisms produce this, and tracing distinguishes between them instantly where metrics cannot:

• The time is in the gaps. Network hops, queue wait, connection acquisition, serialisation. No service's own timer covers those.
• Sequential fan-out. Eight calls of 300 ms each are individually unremarkable and add to 2.4 seconds.
• The tail becomes the norm. A request touching 20 services, each with a p99 of 1 second, has roughly an 18% chance of at least one slow call. The service-level p99 is a page-level p82.
• The aggregate hides the specific. A service's p99 is fine because this particular request was one of the slow ones, and one request does not move an aggregate.
• A retry. Two failed attempts plus backoff before the successful one — invisible in a service's own latency, which measures only successful handling.

A trace shows the tree: where the time was spent, in what order, and what was waiting on what.

What must have been instrumented

Context propagation on every hop, including asynchronous ones. This is where implementations fail: they trace synchronous calls and lose the trace at the queue, which is exactly where the interesting delays are. Trace context must travel in the message envelope.

Spans around the real work, not just around service entry. "Time in the payment service" is not actionable; "2.4 seconds in one query" is. Database calls, external calls, serialisation, cache lookups.

Tail-based sampling. Head-based sampling at 1% means the trace for the incident probably does not exist. Tail-based — keep all errors, all slow requests, and a small sample of normal ones — is worth the extra infrastructure precisely because the whole point is to have the trace for the request that went wrong.

Business attributes on spans — tenant, order ID, plan tier — so you can find traces for a specific customer complaint rather than searching by timestamp.

Links between traces and logs by trace ID, so a suspicious span leads directly to its log lines.

The failure mode to name

Partial adoption. One uninstrumented service in the middle breaks the tree, and it is invariably the legacy one where the time is actually going. A trace that stops halfway is often worse than no trace, because it directs attention to the last instrumented hop.

What a strong answer adds

That tracing tells you where and rarely why. Once the trace points at 2.4 seconds inside one service, the next tool is APM or a profiler to attribute that time internally. Metrics say look; traces say where; profiles say why.