A team computes its service error rate by counting error traces in the trace store. The collector uses tail-based sampling that keeps every trace with an error and 1% of the rest. The true error rate is 0.5% at 10000 requests per second, yet the dashboard reports about a third. Which fix is correct?
Show the full answer Hide the answer
The deciding property
A sampled population is only a measurement of rates if every item had a known and equal chance of being kept. This policy has a different probability for each class: errors are kept with probability 1, healthy traces with probability 0.01. Counting the survivors answers the question "what fraction of stored traces are errors", not "what fraction of requests failed".
The arithmetic
At 10,000 requests per second with 0.5% errors: 50 error traces per second, all kept; 9,950 healthy traces, 1% kept, so 99.5 per second. Stored: 149.5 traces, of which 50 are errors, or 33.4%. The dashboard reports the sampling policy rather than the service. When errors increase to 1%, the figure becomes 50%, so the metric is also non-linear and cannot be alerted on sensibly.
Why derive counts before sampling
Count each span as it passes through the pipeline, before the keep-or-drop decision, and emit request count, error count and duration histogram as ordinary metrics (the span-metrics connector pattern). Every request is counted once, the numbers are exact, and they cost a few thousand time series. Sampling then only decides what detail is stored for drill-down.
Why the other options fail
- "Raise the baseline to 10%." The bias shrinks (stored share becomes about 5%) but stays wrong, and storage cost rises tenfold for a number that is still not the error rate.
- "Multiply the stored error count by the inverse of the baseline rate." Errors are kept at 100%, so scaling them by 100 inflates the count a hundredfold. The correction works only per class: errors by 1, healthy by 100. At that point you are reconstructing counts from weights, which is fragile when the policy changes.
- "Query a longer window." Bias is not noise. More data gives the same wrong ratio with more confidence.
What it costs
The metrics stage sits in the collector tier in front of the sampler, which needs full ingest throughput there. Tail sampling already requires holding all spans of a trace until the decision, so the incremental cost of counting is small compared with the buffer.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| sampling is uniform at 1% and unbiased | scale stored counts by 100 | every request had equal chance, error is statistical |
| the service already emits request metrics from the SDK | use them | pre-sampling counts already exist |
| traffic is under 50 requests per second | keep 100% of traces | no sampling, no bias |
When this is the wrong answer
Pipeline-derived metrics miss spans dropped before the collector (agent buffers, network loss), so cross-check them against load balancer counts occasionally.