Observability Platform  ·  View 16 of 25  ·  Runtime

Tail Sampling

How 2.5 million offered spans per second become 150,000 retained, and what the choice costs.

Editable source SVG draw.io All views
Arrive Spans 2.5 M/s offered Head sample floor volume control only Shard Load-balancing exporter consistent hash on trace id Trace affinity all spans, one shard Hold In-memory trace buffer 45 s window Memory ceiling sized on 5× burst Decide Error status always keep Above service p99 always keep Debug flag set always keep Rare key combination dynamic rate Per-service keep ceiling a keep rule is not a free path Stamp Effective rate written on the trace Keep reason why this one survived Emit Whole trace to the buffer ≈150 k spans/s retained Dropped, counted, attributed reduction ratio published hold expired over ceiling Tail Sampling — Selecting the Evidence Worth Keeping Application we own Decision point Interface / broker Queue / topic Risk / gap failure / alternate synchronous The hold window is the whole cost of tail sampling: it buys the error trace and pays for it in memory at the worst burst, and it is why trace freshness carries 45 s the metric path does not. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Head sampling is kept as a floor for volume control; tail sampling does the evidence selection. Two mechanisms, two purposes, declared separately.
  • A trace kept by the tail sampler is kept whole. Trace affinity is achieved by consistent hashing on trace id before the sampler, because a half-trace is worse than no trace.
  • Keep rules carry a per-service ceiling. Without it, "always keep errors" is an unbounded ingest path that fires hardest exactly during an incident.

What the hold window costs

  • 45 seconds of in-flight spans held in memory, sized against a 5× burst. That is the entire price of tail sampling, and it is why trace freshness carries 45 s that the metric path does not (p95 ≤ 75 s to queryable).
  • The assumption behind 45 s is that it covers p99.9 of in-house request duration. If long-running requests are common, either the window grows or their traces are systematically incomplete.

Risks

  • The sampler is the only memory-bound, effectively stateful tier in the ingest path, and the one component whose capacity must be sized against the worst burst rather than the average.
  • Dynamic rarity-based sampling makes the effective rate vary by key. Every derived count must read the stamped rate, and any consumer that forgets is wrong in a way that looks plausible.
◀  Critical Flow — Sample to Page All views Critical Flow — The Pivot  ▶
Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views16-tail-sampling