An observability bill has grown to a significant fraction of infrastructure spend. Diagnose it systematically and describe the reductions that do not lose diagnostic power.

Diagnose before cutting

Attribute the cost first, by signal type (logs, metrics, traces), by service, and by team. The distribution is almost always extreme: a small number of services or a handful of metrics generate most of the cost, and uniform cuts damage everyone to fix a problem caused by a few.

Then find the specific drivers:

• Cardinality explosions in metrics. One metric with an unbounded label — user ID, request ID, URL path with identifiers embedded — can outweigh everything else combined. List metrics by series count; the answer is usually visible immediately.
• Debug logging left enabled in production, frequently from an incident months ago.
• Duplicate telemetry: the same information as a log, a metric and a span, all retained.
• High-volume logs with no consumer. Access logs already covered by metrics, health-check logs, framework chatter.
• Retention uniformly long, when almost all queries look at the last few days.
• Traces sampled at 100% because it was the default.

Reductions that preserve diagnostic power

• Fix cardinality rather than dropping the metric. Remove the unbounded label, bucket continuous values, normalise paths. The metric stays useful and its cost falls by orders of magnitude.
• Tiered retention: full fidelity for days, downsampled aggregates for months, and long-term retention only for the small set needed for capacity planning and compliance. Query patterns overwhelmingly favour recent data, so this is nearly free in diagnostic terms.
• Non-uniform sampling that keeps all errors and slow requests and samples routine successes heavily. This is the single largest saving available with the least loss, because the discarded data is the data nobody looks at.
• Delete logs that duplicate metrics. If a per-request log line exists only to be counted, the count is the metric and the line is redundant.
• Drop health-check and framework noise at the collector, not at the source, so it can be re-enabled without a deploy.
• Move rarely-queried logs to cheaper storage with slower retrieval.
• Convert repeated log queries into metrics or recording rules, which is cheaper and faster than scanning.

The reductions that look attractive and are not

• Cutting retention uniformly to a few days, which breaks incident investigation for anything discovered late and comparison against previous weeks.
• Uniform sampling, which discards rare events preferentially — exactly the wrong data to lose.
• Removing traces entirely, which removes the only tool that explains latency across services.
• Turning off telemetry for low-traffic services, which are often the ones nobody understands.

Making it stick

Cost must be attributed to the teams that generate it and visible in their own dashboards, or it is a platform problem that no individual team is incentivised to fix. Per-team telemetry budgets with a self-service way to see consumption change behaviour far more effectively than a central cleanup, which decays within two quarters.

Add cardinality limits enforced at ingestion, with a clear error, so a new unbounded label fails fast rather than arriving as a surprise invoice.

The framing for the business conversation

Observability spend should be justified against reduced downtime and faster resolution, not minimised in the abstract. The reasonable question is not "why do we spend this much?" but "what would we be unable to diagnose if we spent less, and what is that worth?" — which converts an argument about cost into an argument about risk, where it belongs.