An edge platform handles enormous request volume and cannot retain telemetry for every request. Which sampling strategy should it use, and what breaks under naive uniform sampling?

What breaks under uniform sampling

The interesting requests are lost. At a 1% rate, an error occurring in 0.01% of requests is almost certainly not captured. The requests you most want are the rarest, and uniform sampling is precisely indifferent to how interesting a request is.

Investigations fail. An engineer looking for a specific customer's failing request will not find it, which destroys trust in the tooling.

Rare-but-severe conditions become invisible, and they are frequently the ones that matter — a failure affecting one region, one device type or one large customer.

Why "errors only" also fails

You lose the baseline. Without normal requests you cannot tell whether a latency profile is unusual, cannot compute meaningful percentiles, and cannot compare a failing path to a working one. Diagnosis requires the contrast.

Why "retain everything, reduce retention" fails at this scale

The cost is dominated by ingestion and indexing, not by retention duration. Shortening retention from thirty days to seven barely helps if the problem is the write path — and it removes the ability to investigate anything historical.

The strategy that works

Tail-based sampling. Buffer the spans of a trace until it completes, then decide whether to keep it based on what actually happened: error, slow, from a customer of interest, or randomly selected as a baseline sample.

The essential properties:

• Consistent decisions across the whole trace. A partially-sampled trace is worse than no trace, because it looks complete and is not. The decision must apply to every span.
• Retained sampling rate on the record, so counts can be scaled back to true values. Without it, sampled data cannot be used quantitatively at all — a common and silent error.
• Metrics computed before sampling. Rates, errors and duration percentiles come from unsampled counters and histograms; sampling applies only to the detailed traces. This is what preserves statistical accuracy while reducing detail volume.
• Per-customer or per-route rate adjustment, so a low-traffic but important route is sampled more heavily than a high-volume health check.

The cost of tail-based sampling

It requires buffering every trace until completion, which needs memory and a collector tier sized for peak throughput. That is a real infrastructure cost, and it is the reason head-based sampling persists.

The pragmatic middle ground is head-based sampling at a generous rate for baseline, with a forced-keep mechanism for requests that mark themselves interesting early — an error already detected, a debug header, a known-problematic customer. It captures much of the benefit without the buffering tier.