beginner 2 min answer Multiple choice

Traces are kept for 1% of requests. Separately the logging agent samples log lines at 1% of its own accord. A customer reports one failed request and you go looking for it. What happens?

samplingcorrelationtrace-contextlogsbeginner
Pick one
Show the full answer Hide the answer

The arithmetic

Two independent decisions multiply. 0.01 × 0.01 = 0.0001, so one request in 10000 keeps both its trace and its logs. At 2000 requests per second that is still 0.2 requests per second with full telemetry, which sounds survivable right up to the moment you want one named request, where the odds are 1 in 10000 against you. The request in the stem also failed, and failures are rarer than 1% of traffic, so the conditional odds are worse again.

The mechanism people miss

Nothing links the two coin flips. A trace ID in a log line identifies a record; it does not cause the record to be kept. A sampler that was never told the trace decision cannot honour it, and two samplers configured with the same rate are still two different random draws.

What to change

  • Make one decision and propagate it. The W3C trace context carries a sampled flag in the traceparent header, and an OpenTelemetry log record has TraceId, SpanId and TraceFlags as first-class fields, where TraceFlags is that flag. The log pipeline then keeps a line when the flag is set and samples routine lines otherwise, so the surviving logs are exactly the surviving traces.
  • Rely on the determinism the specification guarantees. The probabilistic sampler is specified to decide from a hash of the trace ID, so every service in every language reaches the same verdict for the same trace with no coordination, and a service running a higher keep rate retains everything a lower-rate service would have retained.
  • Exempt severity from sampling. Keep every WARN and ERROR record unconditionally. A head decision is made before the outcome is known, so a severity rule that ignores the sampler is the only protection for a request that turns out to fail.

What this still cannot do

It does not rescue the trace. With a 1% head decision, 99 of every 100 failing requests have no trace at all, and no downstream cleverness recovers one. Moving the decision to the end of the trace is the only fix, and it costs a buffering tier holding every in-flight trace, which is a real system to run rather than a configuration change.

When sampling is the wrong answer

Below a few hundred requests per second, keep everything for 7 days. Sampling there saves a small bill and costs you every investigation, because the odds against a specific request are the whole problem and low volume removes the reason to take them. Turn sampling on when the unsampled volume is a line item someone has queried, not by default.