Sampling Cost Curve

A platform cuts trace sampling from 100% to 2% and the observability bill falls by a third rather than by 98%. Nothing is misconfigured. The storage and indexing cost scaled with the keep rate; the collection, transport and decision cost did not, because every span was still produced, serialised, shipped and buffered before anything was discarded.

The curve has three regions, and knowing which one you are on decides whether a sampling change is worth making.

Why it matters

Head sampling is cheap and blind; tail sampling is expensive and correct. Deciding at the start of a trace costs nothing and throws away the interesting traces at the same rate as the boring ones. Deciding after the trace completes can keep every error and every slow request - and it requires holding all spans of every in-flight trace somewhere until the decision is made.

That buffer is the cost nobody forecasts. Memory required is roughly span rate x average span size x trace duration x safety factor: at 200,000 spans per second, 500 bytes a span and a 3-second trace window, that is about 300 MB of live buffer per collector replica before headroom, and it must be sized for the worst trace duration rather than the average. The saving moves from the vendor's invoice to your own collector fleet, and the second is often cheaper - but it is not free, and it is operationally yours.

Implementation patterns

• Bias the sample instead of lowering it uniformly. Keep every error and every request over a latency threshold, sample routine successes at 1 in 1,000. The expensive data is the rare data, and it is a tiny fraction of volume.
• Record the sample rate on every retained item, so counts can be re-weighted. Without that field every derived count silently understates volume.
• Sample per trace or per request, never per span or per log line, or you retain fragments that are worse than nothing.
• Reduce before the keep decision where the cost is collection-side: drop high-cardinality attributes, shorten span names, stop emitting debug spans in production. This is the only lever that moves the flat part of the curve.
• Tier rather than sample for anything used as evidence. An indexed recent window plus a low-compute columnar archive keeps everything and still cuts cost.
• Model the curve before changing the rate: split the bill into ingest, index, store and query, and check which of those actually scales with the keep rate in your contract.

Industry example

Vendor architecture has converged on separating these costs. Datadog's writing on Husky describes decoupling storage from compute and limiting compute for queries that do not need low latency, which is the same insight applied to the query side: pay for the part of the pipeline the question actually needs. Honeycomb's published approach to dynamic sampling takes the other end - derive the keep rate from how common the event is, so a newly rare condition is retained at a high rate without anyone configuring it. Both are responses to the same curve.

Failure scenarios

• A sampling change that saves little, because the bill was dominated by a per-host or per-ingested-byte charge that the keep rate does not touch.
• Collector memory exhaustion when one slow dependency makes traces last 60 seconds and the tail-sampling buffer was sized for 3.
• Unweighted counts used in a capacity plan, understating traffic by the sample factor.
• Error traces lost because sampling is uniform, discovered during the incident the traces were kept for.
• Cardinality smuggled in as attributes, so a 2% keep rate still produces an expensive index because each retained span carries a user id.

Trade-offs

When not to use it

Do not sample data that is evidence. Audit logs, financial transaction records and security events are read for a specific event rather than a distribution, and a sampled audit trail is not an audit trail. Do not sample at all below a few hundred gigabytes a month, where the whole bill is smaller than the engineering time to tune the rules. And do not reach for tail sampling before measuring the curve: if the flat part dominates - per-host agent pricing, per-byte ingest - the cheaper intervention is to emit less, not to keep less.

Interview question

Q: Your tracing bill is £40k a month and you are asked to halve it. Walk me through what you would change, in order, and what you would refuse to change.

What a strong answer covers: splitting the bill into ingest, index, store and query before touching the keep rate; reducing emitted volume and attribute cardinality first because that moves the flat part of the curve; biased rather than uniform sampling, with the sample rate recorded; the collector memory arithmetic if tail sampling is proposed, sized on the worst trace duration; refusing to sample audit and security data; and setting a floor on error and slow-request retention so the next incident is still explicable.

Quick check

Quiz: Why does dropping from 100% to 2% sampling not cut the bill by 98%? Collection, transport and the sampling decision are paid before data is discarded, so only the storage, index and query components scale with the keep rate.

Flashcard: Tail-sampling buffer size? Roughly span rate x span size x trace duration x a safety factor, sized on the worst trace duration - 200k spans/s at 500 bytes over 3 s is about 300 MB per replica.