Trace Sampling Strategy
also called Head Sampling, Tail Sampling
Deciding which traces to keep, since retaining all of them at scale is unaffordable and retaining a random few loses exactly the interesting ones.
Full trace retention is impractical past moderate volume, so the question is which traces survive.
Head-based sampling decides at the start of the request, before anything is known about it. It is cheap and stateless, propagates cleanly through the call chain, and by construction keeps a random sample — which means the error you want to investigate was probably discarded.
Tail-based sampling buffers spans until the trace completes and then decides with knowledge of the outcome, so it can keep every trace that errored, every trace slower than a threshold, and a small percentage of normal ones. That is exactly the right set, and it costs a component that buffers all spans until completion, with memory and complexity proportional to throughput and trace duration.
The strategy that most teams land on: tail-based where the volume justifies the infrastructure, with rules keeping all errors, all slow traces, a low percentage of successful ones, and elevated rates for low-traffic endpoints that would otherwise never appear.
Two details that matter. The sampling decision must be consistent across the whole trace, or you collect fragments that cannot be assembled. And traces should be sampled while metrics remain complete — the aggregate rate and latency come from metrics, which are unsampled, so sampling affects what you can investigate but never what you can measure.