An observability platform's own telemetry costs have grown faster than its revenue. Which decisions reduce it without losing the ability to debug?

What actually drives the cost

Indexing, not storage. The index is often larger than the data, must sit on fast storage to be queryable, and is built at ingestion regardless of whether anyone ever searches it. Raw bytes are cheap.

And cardinality. A metric's cost is driven by distinct series count, so one label with a thousand values multiplies cost by a thousand. The write succeeds, the dashboard renders, and the consequence arrives weeks later.

The decisions that reduce cost without losing debuggability

• Index a chosen subset of fields and store the rest. Searching unindexed data is slower and vastly cheaper, which is the correct trade for anything queried monthly.
• Tier by age: recent data fully indexed, older downsampled and unindexed, oldest in cold storage. Query patterns follow recency very strongly, so this is close to free in usefulness terms.
• Tail-based sampling for traces, keeping every error and every slow request plus a small random baseline — which retains almost everything worth having at a fraction of the volume.
• Move what is really a metric out of logs. Counting log lines to derive a rate is enormously more expensive than emitting a counter, and it is a very common pattern.
• Per-signal, per-team retention, since a debug log and a compliance record have nothing in common and one policy over-retains one and under-retains the other.
• A cardinality budget per team, measured and alerted, which is the only control that puts the cost in front of the person creating it.

What must not be cut

Logging on the critical debugging path. The signals that matter are the ones present during an incident, and cutting them buys money in exchange for longer outages — an easy trade to make accidentally when cutting by volume rather than by value.

And nothing at all before measuring what is queried. Cutting by guesswork removes what someone needed and keeps what nobody wanted. Instrumenting the query patterns is a day of work and is the cheapest input to every subsequent decision.

The control that changes behaviour

Cost attribution per team and per service, made visible. Telemetry volume is created by developers who currently see no cost signal, and no technical control changes that as effectively as showing them the number.