A team enabling distributed tracing must decide between head-based and tail-based sampling. Compare them, and explain what makes traces useful beyond a single request's timeline.

Head-based sampling

The decision is made at the start of the trace — typically at the first service — and propagated, so every service honours the same choice and the trace is complete or absent.

Advantages: simple, no buffering, no additional infrastructure, and the decision costs nothing downstream.

Disadvantage, and it is the important one: the decision is made before the outcome is known. A request that turns out to be slow or to fail was sampled at 1% along with everything else, so the interesting traces are discarded at the same rate as the boring ones.

Mitigated by sampling on request attributes available at the start — endpoint, customer tier, a debug header — and by forcing sampling for specific conditions, but never fully solved, because the outcome is not knowable at the start.

Tail-based sampling

The decision is made after the trace completes, so it can consider the outcome: keep all errors, keep everything above the latency threshold, sample successes heavily.

Advantage: exactly the right semantics. You keep what is interesting and discard what is not.

Cost: every trace must be buffered until it completes, which requires a collector tier holding all in-flight traces, with memory proportional to trace volume × duration. All spans of one trace must reach the same collector instance, which means consistent routing by trace ID and a real distributed system to operate. Late spans complicate the decision, and the collector becomes a component that can lose data.

The practical answer

Head-based with intelligent rules for most organisations; tail-based when the volume and the diagnostic need justify the infrastructure. A common and effective hybrid: head-based sampling with a forced-keep rule for errors, plus a debug header that forces a full trace on demand — which covers most investigative needs at a fraction of the operational cost.

What makes traces valuable beyond a timeline

The waterfall view is the least interesting thing traces provide.

• The service dependency graph, derived from actual traffic rather than from a diagram someone drew. This is frequently the only accurate map of the system in existence.
• Aggregate analysis across many traces: which dependency contributes most to p99 latency, which service causes the most errors in a journey, how the graph differs for slow requests versus fast ones.
• Trace-derived metrics — RED metrics per service, computed from spans, consistent across every service without each team instrumenting separately.
• Critical path analysis, distinguishing time actually blocking the response from parallel work that does not — which is where the optimisation effort should go, and is invisible in a per-service latency metric.
• Correlation with logs and metrics via the trace ID, which is what turns three separate tools into one investigation.

The prerequisite everyone underestimates

Context propagation must be complete. One service that drops the trace context breaks every trace passing through it, and the breakage is silent — traces simply appear to end. Async boundaries are where this fails most often: message queues, background jobs, thread pools and event handlers all need explicit context propagation, and an incomplete trace is frequently more misleading than no trace at all, because the missing segment looks like an absence of work rather than an absence of instrumentation.