Distributed Tracing in Practice

Definition

A trace is the tree of operations produced by one logical request. Each unit of work is a span with a start, a duration, a parent, and attributes. Context propagates across every hop through headers.

What it answers that nothing else does

Where the time went. A request took 3 seconds: 40 ms in the gateway, 60 ms in the API, 2.7 s waiting on one downstream call which itself spent its time waiting on a database. Reconstructing that from logs across five services is hours of work and is usually not attempted.

What the actual dependency graph is. Not the diagram — what the code really calls, including the call nobody documented.

Where the fan-out is. A trace showing 340 sibling spans to the same service is an N+1 pattern made visible, and it is the fastest way to find one.

Sampling, which is the operational decision

Tracing every request is prohibitively expensive at volume. Three strategies:

• Head-based, deciding at the start — simple, and it discards most of the interesting requests because errors and slow requests are rare.
• Tail-based, deciding after completion — keep all errors, all slow requests, and a small sample of normal ones. Far more useful, and it requires buffering complete traces somewhere before the decision.
• Adaptive, varying by endpoint so a low-traffic critical path is sampled more heavily than a high-traffic trivial one.

Tail-based sampling is worth the extra infrastructure for most systems, because the whole point is to have the trace for the request that went wrong.

Making it actually useful

• Propagate context across asynchronous boundaries. Most implementations trace synchronous calls and lose the trace at the queue, which is exactly where the interesting delays are.
• Add business attributes to spans — tenant, order ID, plan tier — so you can find traces for a specific customer complaint.
• Link traces to logs by trace ID, so a span leads directly to its log lines.
• Instrument the database call, not just the service call. "Time in the payment service" is not actionable; "2.4 s in one query" is.

Failure scenarios

• Partial adoption. One uninstrumented service in the middle breaks the tree, and it is invariably the legacy one where the time is going.
• Head-based sampling at 1%, so the trace for the incident does not exist.
• Traces retained for days when incidents are investigated over weeks.
• Spans without attributes, so you can see that something was slow and not which thing it was.
• Trace context dropped by a proxy or gateway that does not forward the headers.

Interview question

"A request is slow but every individual service reports healthy latency. How does tracing resolve that, and what would you have needed to instrument?"