Cardinality Budget

A metric's cost is driven by distinct series count, not by write frequency. Adding one label with a thousand values multiplies the series count by a thousand, and series count determines index size, memory footprint and query cost.

The failure is silent by construction: the write succeeds, the dashboard renders, and the consequence appears weeks later as an invoice or a query timeout. One thoughtless label — user ID, request ID, full URL path, container ID — can multiply an organisation's telemetry cost by orders of magnitude overnight.

Why it matters

Nothing else works. Education does not scale, review does not catch it, and the person who adds the label is not the person who receives the bill. A budget is the only mechanism that puts the cost in front of the team creating it, at the time they create it.

Implementation patterns

• A per-team or per-service series budget, measured continuously and alerted on before it is exceeded.
• Reject high-cardinality labels at ingestion, with a clear error naming the offending label. A silent accept is how the problem becomes permanent — once dashboards depend on it, removal is a negotiation.
• Separate signals by what they are good at. Metrics for low-cardinality aggregates; traces for high-cardinality context, sampled; logs for full detail, cheap to store and expensive to index. Trying to make metrics answer high-cardinality questions is the root cause of most cost blowouts.
• Exemplars — metrics that link to representative traces — which allow drill-down into high-cardinality detail without carrying the cardinality on the metric itself. This resolves most of the tension.
• Top-N plus "other" aggregation for dimensions like tenant, which gives visibility where it matters and bounds series count, and works because the distribution is heavily skewed.
• Show cost attribution per team, since the budget is only credible if the underlying number is visible.

Industry example

Observability vendors such as Grafana Labs and Elastic encounter this from both sides: their own platforms must survive customers' cardinality, and their customers must be helped to control it. The product feature that matters most in practice is not a better index but a per-tenant series limit with clear feedback, because it is the only thing that changes the behaviour producing the cost.

Multi-tenant SaaS platforms hit the same wall internally when they add tenant identity to every metric — a change that is obviously desirable for attribution and multiplies series count by the tenant count.

Failure scenarios

• An unbounded label added in a routine change, discovered on the invoice.
• High-cardinality labels accepted silently, becoming load-bearing before anyone notices.
• Metrics used for high-cardinality questions, when a trace or a log would answer them at a fraction of the cost.
• A budget with no owner, so exceeding it produces an alert nobody acts on.
• Cutting telemetry by guesswork rather than by measured query patterns, removing what someone needed and keeping what nobody wanted.

Trade-offs

Every cardinality control reduces what can be asked later. Dropping a label loses a breakdown; the question it would have answered now requires a new deployment and a wait for data.

The honest resolution is to instrument the query patterns and cut what nobody queries. That requires knowing what questions are actually asked — which most organisations have never measured, and which is the cheapest possible input to this decision.

Interview question

"Your telemetry bill tripled last quarter with no traffic growth. Walk me through how you would find the cause, and then tell me what control would have prevented it without stopping teams from instrumenting their services."