Observability Cost

Definition

Logs, metrics, traces and their storage, indexing and querying. It is one of the few costs that scales exactly with success, and at scale it commonly reaches 10–30% of infrastructure spend, occasionally more.

Where the money goes

• Log volume and indexing. Usually the largest. Indexing is far more expensive than storage, and most logs are never queried.
• Metric cardinality. Cost scales with unique label combinations, not with traffic. One high-cardinality label — a user ID, a request ID, a URL with parameters — can multiply the bill and destabilise the backend.
• Trace volume, and the retention window.
• Per-host agent licensing in commercial tools, which scales with fleet size rather than value.
• Long retention on everything, applied uniformly because nobody classified.

The levers

• Sample logs and traces, keeping 100% of errors and slow requests. Sample by trace rather than by line, so a kept request retains its whole story.
• Audit metric cardinality and remove unbounded labels. Frequently the single largest saving, and it improves stability at the same time.
• Tier retention by class. Days hot and searchable, months in cheap object storage. Audit logs on their own long retention; debug logs on days.
• Move counters out of logs into metrics, which is far cheaper per event.
• Delete unused dashboards, alerts and metrics. Most estates emit large numbers of metrics nobody has ever queried.

The judgement required

The failure mode at both extremes is real. Cutting observability to save money removes the ability to diagnose incidents, and a longer outage costs more than a year of the tool. Retaining everything forever is a large recurring bill for data nobody reads.

The right frame is per-signal: what question does this answer, and how often is it asked? A signal nobody has queried in six months is a candidate for deletion; a signal used in every incident is worth whatever it costs.

Failure scenarios

• Unbounded metric cardinality taking down the metrics system and the budget together.
• Uniform retention, so debug noise is kept as long as audit records.
• Sampling introduced during a cost cut, then discovered to have removed exactly the traces needed during an incident.
• Per-host licensing on a fleet that scaled horizontally, so the tool cost tracks instance count rather than value.

Interview question

"Your observability bill is 25% of your infrastructure spend. How do you reduce it without losing the ability to diagnose incidents?"