A platform with extreme traffic spikes needs distributed tracing. Head-based sampling loses the interesting traces and 100% sampling is unaffordable. What strategy resolves this?

Why head-based sampling fails

The decision is made when the trace starts, before anything is known about it. A 1% head sample keeps 1% of errors and 1% of slow requests — which is to say it keeps almost none of the traces anyone will ever want, and it keeps a great many uninteresting successful ones.

During a spike it is worse: the absolute number of interesting traces rises exactly when the sample rate is most likely to have been reduced to control cost.

Tail-based sampling

Buffer the complete trace, decide after it finishes. Keep every trace that errored, every trace above a latency threshold, every trace touching a rare code path, plus a small random sample of normal ones for baseline comparison.

This keeps almost everything worth having at a fraction of the cost. The price is real: the collector must buffer all spans of all in-flight traces until each completes, which requires memory proportional to throughput times trace duration, and it needs all spans of a trace to arrive at the same collector — which is a non-trivial routing constraint at scale.

The layered approach that works in practice

• Head sampling at a low rate for baseline, cheap and always on.
• Tail sampling for the interesting population, with rules by error, latency, endpoint and customer tier.
• Always sample specific things fully: a named debugging session, a specific customer under investigation, a newly deployed code path, anything above a value threshold.
• Client-controlled forcing, so a support engineer can request full tracing for one user's next request.
• Metrics for aggregates, always at 100%. Sampling is for traces; error rates and latency percentiles must be computed from complete data or they are wrong in a way nobody notices.

The mistake that makes sampling useless

Sampling per service rather than per trace. If each service samples independently, traces are fragmented — you have the middle of one request and the start of another, and neither reconstructs. The sampling decision must propagate through the trace context, which is the single most important implementation detail and the one most often got wrong in a partial rollout.