Your observability bill is now 40% of your compute bill. Leadership wants it cut in half without going blind. What do you cut?

What the interviewer is testing

Whether you understand what each telemetry type is for, so you can cut the redundant parts rather than cutting uniformly — which is the response that does go blind.

First: find where the money is

Observability spend is almost always dominated by one or two of: log volume, metric cardinality, trace volume, and retention duration. Break it down before touching anything — the answer is usually concentrated, and it is usually not where people assume.

The cuts, in order of return per unit of risk

1. Metric cardinality. Frequently the largest and most easily fixed. One label with an unbounded value — user ID, request ID, full URL with parameters, error message text — multiplies series count by its cardinality. Find the top series-producing metrics and remove the offending labels. That information belongs in a log line or a trace attribute, where it costs far less. This alone often delivers a large fraction of the target with no loss of capability, because those series were never queryable in aggregate anyway.

2. Log level and volume. Debug logging left on in production is a classic. Also: logs emitted per iteration inside loops, full request and response bodies, and health-check access logs, which in a Kubernetes cluster can be the single largest log source and carry no information.

3. Retention, tiered by type. Very little is queried after 30 days, and almost nothing after 90 — but compliance may require longer. The pattern: hot searchable storage for 7–30 days, cheap object storage for the rest, restored on demand. Retention is usually the cheapest large saving because it needs no instrumentation change.

4. Tail-based sampling for traces. Do not cut trace coverage, cut trace retention rate intelligently: keep 100% of errors and slow requests, and 1% of fast successful ones. Head-based sampling at a flat 1% would discard the traces you actually need; tail-based keeps them. This is a capability improvement disguised as a cost cut.

What not to cut

• The golden signals for every service. This is the smallest set that tells you whether users are being served, and it is cheap because it is low cardinality.
• Error traces and error logs, in full.
• Audit logs, which are a compliance obligation, not observability spend.

The structural fix

The bill grew unnoticed because nobody owned it. Make telemetry cost visible per team and per service, add cardinality limits at the collector so a new label cannot silently multiply spend, and review instrumentation as part of code review the same way you would review a database query.

What a strong answer adds

Framing it as cost per unit of diagnostic value: the goal is not less telemetry, it is not paying to store telemetry that has never been queried. Most backends can report which metrics and dashboards have not been read in 90 days, and that report is the most defensible cut list there is.