Per-Device Series Budget

A fleet platform emits 25 metric series per device and tags each with a device id. With 400,000 devices that is 10 million active time series, because a metrics system creates one series per distinct label combination and prices, indexes, retains and queries per series. The data volume is unremarkable; the series count is the bill.

The reflex is to drop the device id and keep per-model and per-region aggregates. With 12 models and 6 regions that is about 1,800 series, four orders of magnitude cheaper, and it silently removes the only question the support desk ever asks: what did this one unit do before it went quiet?

Per-device series budget is the number you set deliberately instead of discovering. It is the product of series per device and device count, compared against what the metrics backend can hold at a price you accept, and it forces the real decision: which questions belong in a metrics system, and which belong somewhere cardinality is cheap.

Why it matters

Observability spend at fleet scale routinely exceeds the compute cost of the workload being observed, and cardinality is the reason: cost is multiplicative in label values, not additive in data points. Adding one label with 400,000 values multiplies the series count by 400,000.

The second reason is diagnostic. An average across 400,000 devices is a number no device experiences, and a fleet-wide p99 tells you 4,000 units are unhappy without telling you which, so it generates work instead of answers. Aggregates hide exactly the shapes that matter in a physical fleet: 800 units in one building degrading together, a component batch appearing as a bimodal distribution, one hardware revision behaving differently under a new firmware.

Implementation patterns

• Split the two jobs. Low-cardinality aggregates in the metrics system for alerting and trends; one wide event per device per interval in columnar storage on object storage, where a device id is a column rather than a series and cardinality is nearly free.
• Two retentions. 13 months for aggregates so year-on-year comparisons work; 14 to 30 days for per-device events, extended only for units under investigation.
• Budget by tier. Fleet-wide series capped at a few thousand; per-device detail unlimited in the event store; a named canary cohort of a few hundred devices allowed full metric series so regressions are caught with normal alerting.
• Sample by interest. Every device emits a heartbeat and a small vector; devices that are erroring, recently updated or in the canary cohort emit everything.
• On-demand pull. A local ring buffer on the device, uploaded when asked. Cheapest of all, and it fails exactly when the device is unreachable, which is when the data is most wanted, so it supplements rather than replaces the event stream.
• Enforce it in code. A metrics wrapper that rejects unbounded label values at review or runtime, because the regression is always one engineer adding one useful tag.

Industry example

The industry moved this direction publicly. Observability vendors built columnar event stores precisely because per-series indexing does not survive high-cardinality fields: Honeycomb's design writes wide events to column stores and scans them, and Datadog described Husky, its event store on object storage, in 2022 for the same reason. The shared conclusion is that a high-cardinality identifier should be a column, not a series — a metrics system's cost model cannot be argued with, only avoided.

Failure scenarios

• A silent bill step-change after someone adds a firmware-version or site-id tag, which multiplies rather than adds.
• Query timeouts on dashboards as series counts cross the backend's index limits, usually first noticed during an incident.
• Retention quietly reduced to control cost, which removes the year-on-year comparison that justified the metrics system.
• Diagnosis by guesswork after aggregation, where a 2% regional regression cannot be traced to units and the fix is a firmware rollback for everyone.
• Missing the absent devices. Aggregates report on devices that report; a unit that stopped reporting three weeks ago contributes nothing to any series and shows up in no alert unless absence is modelled explicitly.

Trade-offs

When not to use it

Below a few thousand devices the series count is not the constraint, and budgeting it costs you diagnosis for a saving nobody asked for. Tag everything, keep the device id, and revisit at the order of magnitude where the metrics bill becomes visible next to the workload. The budget also does not apply if your backend genuinely prices on ingested samples rather than active series, which a few do — check the pricing model before designing around a constraint you may not have. And a canary cohort should always be exempt: a few hundred devices with full detail is the cheapest early-warning system available.

Interview question

Q: Your fleet's metrics bill has passed the cost of running the platform itself. A colleague proposes removing the device id tag. Argue both sides, then tell me what you would actually do and what you would lose.

What a strong answer covers: that cost is multiplicative in cardinality, with the arithmetic; the specific diagnostic capability lost, and that the loss is retrospective so it lands on the first non-fleet-wide incident; the split design — aggregates for alerting, wide events in columnar storage for per-device history, two retentions; a canary cohort as the exception; and the scale threshold below which none of this is worth doing.

Quick check

Quiz: Why does adding a device id tag cost more than adding a device? Because each distinct label combination is its own time series, so the tag multiplies the series count by the fleet size rather than adding to it.

Flashcard: Where should per-device telemetry live once a fleet passes tens of thousands of units? — In wide events in columnar storage where the device id is a column, with the metrics system keeping only low-cardinality aggregates for alerting and trends.