You are introducing distributed tracing across 40 services owned by 12 teams. Plan the adoption.
Show the full answer Hide the answer
The property that shapes the whole plan
Value is non-linear in coverage. One service that does not propagate context breaks the trace into fragments, and everything downstream becomes unattributable. Ninety percent coverage delivers far less than ninety percent of the benefit.
So this cannot be a per-team opt-in with a long tail. It has to be driven as a platform programme with a coverage target.
Sequence
Auto-instrumentation first. For common frameworks and clients it produces useful traces with no code changes — usually 80% of the value for a small fraction of the effort, and it gets breadth quickly, which is what the non-linearity demands.
Propagation before enrichment. Get W3C Trace Context flowing across every boundary before anyone adds custom spans. Use the standard rather than a bespoke header so vendor and third-party components participate.
Then manual spans and attributes where automatic instrumentation is insufficient — business operations, in-process work that dominates a request, and domain identifiers as span attributes.
Where propagation actually breaks
Thread and coroutine handoffs — context in thread-local storage is lost when work moves to a pool. Frameworks provide context-aware executors; using them is not automatic.
Asynchronous messaging — the producer must write context into message headers and the consumer must restore it. This is the most commonly missed case, and it is where the most valuable traces live, because asynchronous causation is the hardest thing to debug without them.
Scheduled jobs and batch processes, which have no incoming request and need a context created deliberately.
Third-party and legacy components that drop unknown headers, which fragments traces in a way that looks like an instrumentation bug.
Sampling, decided before volume becomes a problem
Tail-based sampling: keep all errors, everything above a latency threshold, and a small baseline of successes. Head-based sampling at 1% discards 99% of exactly the traces anyone would look at.
Its cost is a real constraint and the usual reason it is not enabled: the collector must buffer all spans for a trace, so all spans for one trace must reach the same collector instance — collectors must be load balanced by trace ID.
And sampling decisions must propagate in the trace context, or traces fragment as each service decides independently.
Make it worth adopting
Adoption stalls when teams see cost and no benefit. Give them something immediately: service dependency maps derived from real traffic (which reliably differ from the documented architecture and are instantly useful), latency breakdown by hop, and exemplars linking metrics to representative traces so the tools compose.
Then demonstrate a real diagnosis — take a recent incident that took hours and show it resolved in minutes with traces. That single demonstration moves adoption faster than a mandate.
Governance
Semantic conventions for span and attribute naming, so traces from different teams are comparable. Coverage tracking — percentage of services propagating, and percentage of traces that are complete — published, because the non-linearity means the gaps matter more than the average.
No personal data in span attributes. Traces are exported to third-party backends and retained.