A platform's telemetry spend approaches its compute spend. Which reduction preserves diagnostic value, and which destroy it?
Show the full answer Hide the answer
Reductions that preserve or improve diagnostic value
1. Tail-based sampling. Keep all errors and slow requests plus a small baseline; discard ordinary traces. This improves diagnostic value while cutting volume, because uniform sampling loses exactly the requests worth investigating. It is the rare cost control that is not a trade.
2. Moving counting out of logs into metrics. Much log volume exists so someone can count occurrences. A metric is orders of magnitude cheaper, faster to query, and retained longer.
3. Edge aggregation. Pre-aggregating in the agent before data crosses the network drops cardinality at the cheapest possible point, with no loss for aggregate queries.
4. Tiered retention by data class. Errors and audit events retained long, debug telemetry for days, downsampled metrics for capacity planning. Uniform retention is the easiest large saving available and costs almost nothing diagnostically.
5. Removing duplicate telemetry — the same information emitted as a metric and a log line and a trace attribute.
Reductions that destroy diagnostic value
1. Uniform trace sampling at a low rate. The specific request under investigation will not be sampled. Engineers try twice, find nothing, and stop using the tool — so the spend that remains buys nothing.
2. Dropping high-cardinality dimensions indiscriminately. Tenant, region and version are precisely the dimensions needed to find a concentrated failure. Cardinality must be controlled by removing unbounded dimensions (request ids, full URLs), not useful ones.
3. Shortening retention below the detection delay. If problems are found six hours later and telemetry is kept four, the evidence is gone. This parameter's correctness depends on detection latency, not on storage cost.
4. Removing business metrics because they are low volume and unfamiliar. They are the cheapest telemetry and the only detector for failures where the system works correctly and produces wrong results.
5. Disabling logging in a high-volume path that is precisely where incidents occur.
The governing question
For each of the top ten volume sources: when was this last queried, and by whom? Observability platforms record their own query patterns, so this is answerable — and sources with no queries in ninety days are deletion candidates regardless of how reasonable they seemed when added.
That question reduces spend without reducing capability, which is the only reduction worth making. The frame is cost per unit of insight, not absolute cost: the goal is not less telemetry, it is not paying for telemetry nobody reads.