An observability platform ingests enormous telemetry volume, but a small number of dimensions create extreme cardinality. How should ingestion, aggregation, indexing, sampling, retention and storage tiers be designed so cost and query performance stay predictable?

Why cardinality dominates cost

A metric's cost is not driven by how often it is written but by how many distinct series it creates. Adding one dimension with a thousand distinct values multiplies the series count by a thousand, and series count determines index size, memory, and query cost. A single dimension added thoughtlessly — user ID, request ID, full URL path, container ID — can multiply a system's telemetry cost by orders of magnitude overnight.

The insidious part is that it looks harmless at write time. The write succeeds, the dashboard works, and the cost appears weeks later in an invoice or as a query timeout.

The design that keeps it bounded

• Enforce a cardinality budget per team or per service, measured and alerted on. This is the control that actually works, because it turns an invisible externality into a visible constraint the team owns.
• Reject or drop high-cardinality labels at ingestion, with a clear error, rather than accepting them and paying. A silent accept is how the problem becomes permanent.
• Separate the three signal types by what they are good at. Metrics are for aggregates and must be low-cardinality. Traces carry high-cardinality context and are sampled. Logs carry full detail and are cheapest to store and most expensive to index. Trying to make metrics answer high-cardinality questions is the root cause of most cost blowouts.
• Index selectively. Store everything, index a chosen subset. Search over unindexed data is slower and vastly cheaper, and for data that is queried once a quarter that is the correct trade.
• Tier by age with different resolutions. Recent data at full resolution and full indexing; older data downsampled and unindexed; oldest in cold object storage. Query patterns follow recency very strongly, so this is close to free.
• Retention set per signal and per team, not globally, since a debugging log and a compliance record have nothing in common.

The trade-off to state honestly

Every one of these reduces what can be asked later. Downsampling loses the spike, sampling loses the individual trace, dropping a label loses a breakdown. The question is not whether to lose fidelity but which fidelity is worth its cost, and that requires knowing what questions are actually asked — which most organisations have never measured.

Instrument the query patterns, then cut what nobody queries. Cutting by guesswork removes the data someone needed and keeps the data nobody wanted.