Telemetry Cost Management

Observability commonly reaches 10–30% of infrastructure spend, and occasionally exceeds the cost of the systems it observes. It grows silently because every team adds telemetry and nobody removes any.

The cost drivers, in typical order: log volume, metric cardinality, trace volume, and retention duration.

Reductions that preserve capability:

Sample logs of successful requests aggressively; retain errors in full. Successful-request logs are the bulk of the volume and are almost never read individually.

Tail-based sampling for traces — all errors, all slow traces, a small baseline of successes.

Cardinality allow-lists enforced at the collector, and an alert on series growth.

Tiered retention — high resolution for days, downsampled aggregates for months, cold storage for compliance. Almost all queries look at the last few days.

Delete unused dashboards and alerts, and drop metrics nothing queries. Most backends can report query usage per metric; the result is usually startling.

The governance that prevents recurrence: per-team cost visibility. Teams that cannot see their telemetry spend have no reason to manage it, and central mandates without visibility simply relocate the argument.