An observability platform lets customers attach arbitrary tags to metrics. One customer adds a tag containing a request identifier. What would you change about the system's constraints, and why is "tell the customer not to do that" an insufficient answer?
Show the full answer Hide the answer
What just happened
A metric with a unique tag value per request is not a metric — it is a log with worse ergonomics. Every distinct tag combination is a separate time series to index, store and query. One tag turned a handful of series into millions, and in a shared platform the cost lands on infrastructure shared with every other customer.
This is a technical constraint that was never expressed in the product. The system permits something it cannot afford.
Why "tell them not to" fails
Three reasons, and they generalise well beyond observability:
- It is not discoverable. The customer did something the API allowed, got no error, and saw correct-looking data. Every incentive said "this works".
- It is not enforceable. There is no way to distinguish "a tag with many values" from "a tag with unbounded values" by inspection at write time — only by watching the trend.
- It scales the wrong way. Guidance requires every current and future customer to read and follow it. A constraint enforced by the platform requires nobody to read anything.
Any control that depends on users behaving well is not a control. It is a hope with documentation.
What to change
Make the constraint explicit and priced. Series count per customer becomes a first-class, visible, billed dimension. Once a customer can see it and pays for it, the incentive aligns without any conversation.
Enforce at ingestion with a budget, not a hard cliff. Per-customer series limits with a soft threshold that warns and a hard ceiling that sheds. Critically, when the ceiling is hit, drop the offending dimension rather than the metric — preserving a usable aggregate is far better than losing the signal entirely.
Detect unbounded dimensions automatically. A tag whose distinct-value count grows roughly one per data point is an identifier, not a dimension. Flag it, surface it to the customer, and offer the correct tool — that data belongs in traces or logs where high cardinality is the design point.
Isolate the blast radius. Per-tenant ingestion quotas and separate indexing capacity so one customer's cardinality explosion degrades their own queries first, not everyone's.
The architectural principle
When a platform accepts arbitrary input from tenants, every dimension of that input needs a bound, a meter and a graceful behaviour at the bound. Unbounded cardinality is the observability-specific instance of a general rule: any input a tenant controls will eventually be maximised, usually by accident.