A marketplace adopts distributed tracing but engineers rarely use it during incidents. What typically causes low adoption, and what makes tracing actually useful?
Show the full answer Hide the answer
Why adoption fails
1. Incomplete propagation. If context is dropped anywhere — a queue, a background job, a legacy service, a third-party library — traces terminate at that boundary. A trace that stops before the interesting part is worse than useless, because it wastes the engineer's time and teaches them not to look.
2. Uniform random sampling. Sampling 1% of requests means that when investigating a specific slow request, it is almost certainly not sampled. Engineers try tracing twice, find nothing, and stop.
3. No path from a symptom to a trace. An engineer sees a latency spike on a dashboard and has no way to get from that spike to an example trace. Searching for one manually is tedious enough that it does not happen under time pressure.
4. Traces without business context. A trace showing service call timings but not which tenant, which operation, or which order cannot answer "why is this customer affected".
5. Slow, awkward query interfaces, where finding the right trace takes longer than guessing.
What makes it useful
Tail-based sampling. Decide whether to keep a trace after it completes, keeping all errors, all slow requests, and a small random sample of normal ones. This inverts the economics: you retain exactly the traces you would want and discard the ones nobody will look at. It costs buffering at the collector, which is a fair price.
Exemplars linking metrics to traces. A dashboard spike is one click from an example trace of a request in that bucket. This is the single highest-leverage integration in observability tooling, because it removes the search step entirely.
Rich span attributes — tenant, operation, entity id, version, region — so a trace answers business questions and can be searched by them.
Complete propagation, enforced. Context passing through message queues and background jobs, verified by a test rather than by convention. Asynchronous boundaries are where propagation is usually lost.
Traces on the critical path first. Full coverage is a multi-year project; the checkout path is a quarter's work and delivers most of the value.
The measure of success
Time to attribute. Before tracing, "which service is causing this" takes hours of correlating dashboards. After, it should take minutes. If it does not, the implementation is incomplete — and the honest response is to fix propagation and sampling rather than to conclude that tracing does not help.