Design telemetry collection where the volume of monitoring data would cost more than the systems being monitored.

Where the cost comes from

Cardinality, not volume. Metric cost scales with the number of distinct time series, which is the product of every label's distinct values. A device identifier as a metric label on a fleet of a million devices creates a million series per metric, and adding a second high-cardinality label multiplies again.

This is the most common cause of unexpected observability cost, and it is usually introduced by a small change that looks harmless.

The controls

• Aggregate at the edge or on the device. Send percentiles and counts per period rather than every reading. A device reporting summarised statistics every minute costs a fraction of one streaming raw values.
• High-cardinality identifiers in logs and traces, not in metric labels. Metrics answer "how is the fleet behaving"; identifying a specific device is a query against event data.
• Tiered retention: raw briefly, rolled up for longer, aggregated indefinitely. Query patterns follow the same shape, so this rarely costs anything anyone notices.
• Sampling with a policy — head-based for volume reduction, tail-based to keep the interesting cases, always keeping errors and slow outliers.
• Adaptive detail: minimal telemetry normally, elevated when a device or region shows anomalous behaviour. This gives the resolution where it is needed without paying for it everywhere.

The decisions that must precede ingestion

Retention and rollup policy. Rollups are far cheaper to define at ingestion than to backfill, and raw data never retained cannot be rolled up differently afterwards. The asymmetry means the policy forecloses options in one direction only, so it deserves attention before the first event rather than after the first bill.

The measure to hold it to

Cost per question answered, not cost per metric collected. Most telemetry is never queried, and the highest-return exercise is identifying what is actually consumed — which requires query attribution that most platforms have and few teams look at.