An observability platform ingests billions of telemetry events daily, and some customers attach dimensions with unbounded distinct values. How should ingestion, storage, indexing and query be designed so cost and latency stay manageable?

Why cardinality is the governing constraint

A time series is identified by a metric name plus its full set of label values. Every distinct combination is a separate series to index, store and query. Cardinality is therefore multiplicative: five labels with ten, twenty, five, a hundred and three values respectively produce 300,000 series from one metric.

Add one label whose values are unbounded — a request id, a user id, a full URL with parameters — and the series count becomes proportional to the event count. At that point it is not a metric system; it is a log system with worse ergonomics and much worse economics.

The design

1. Separate the storage by data shape, not by product name. - Metrics — bounded cardinality, pre-aggregated, cheap, retained long, queried in aggregate. - Traces — high cardinality by design, sampled, retained briefly, queried by identifier. - Logs — unbounded detail, sampled or tiered, queried rarely.

Most cardinality disasters are a shape mismatch: high-cardinality data pushed into the metric path because that is where the dashboards are.

2. Enforce a per-tenant series budget at ingestion. Soft threshold that warns, hard ceiling that acts. Critically, at the ceiling drop the offending dimension rather than the metric — preserving a usable aggregate is far better than losing the signal entirely.

3. Detect unbounded dimensions automatically. A label whose distinct-value count grows roughly one per data point is an identifier, not a dimension. Flag it, surface it to the customer, and route them to the correct tool.

4. Make series count a visible, billed dimension. Once the customer can see it and pays for it, the incentive aligns without any conversation. This does more than any technical control.

5. Isolate the blast radius. Per-tenant ingestion quotas and separate indexing capacity, so one customer's explosion degrades their own queries first rather than everyone's.

6. Aggregate at the edge. The agent pre-aggregates before transmission, dropping cardinality at the cheapest possible point — before it crosses the network and before it reaches the index.

7. Tiered retention with downsampling. Full resolution for hours, downsampled for weeks, rolled up for years. Query cost then tracks the resolution the question actually needs.

The architectural principle

Any dimension a tenant controls will eventually be maximised, usually by accident. Every tenant-supplied input therefore needs a bound, a meter, and a defined graceful behaviour at the bound — and guidance in documentation is not a control, because it requires every current and future customer to read and follow it.