A single deploy took down your monitoring platform. What happened, and how do you prevent a recurrence?

What almost certainly happened

A high-cardinality label was added to a metric.

Each unique combination of label values is a separate time series, and cost scales with the product of the label cardinalities. A metric with 10 endpoints × 5 status codes × 3 regions is 150 series. Adding user_id with a million values makes it 150 million.

The monitoring backend runs out of memory or hits its ingestion limit — and the ingestion pipeline is shared, so all telemetry stops, not just the offending metric. You are now blind, during an incident your own change caused.

The usual culprits are well-intentioned: user ID, request ID, trace ID, session ID, full URL with query parameters, error message text, or a raw timestamp used as a label.

Immediate response

Identify the metric by series count, drop it at the collector, and confirm ingestion recovers. Roll back the deployment if the emission is in application code and cannot be dropped upstream.

Prevention, in layers

Cardinality limits at the collector. Enforce a per-metric series cap that drops the offending metric rather than allowing it to take down the pipeline. This is the control that would have contained this incident, and it is usually not configured.

Label allow-lists — a defined set of permitted label keys, with anything else rejected. Stronger than limits, because it fails at emission rather than after the damage.

Alert on series growth rate, so a cardinality increase pages before it saturates.

Detect in CI. A linter that flags metric labels containing known-unbounded names, and a staging environment where series counts are compared before and after a change.

Isolate ingestion so one team or one metric cannot exhaust shared capacity — bulkheading applied to the monitoring platform, which is production infrastructure and deserves the same treatment.

Fix the underlying misunderstanding

The engineer wanted per-user visibility, which is a legitimate need. Metrics are the wrong tool.

Metrics answer "how many, how fast, how often" in aggregate. Traces answer "what happened to this particular request". Logs answer "what happened in detail". High-cardinality data belongs in the latter two, where the storage model is designed for it.

Publishing that rule — with the safe and unsafe label lists — prevents more recurrences than any technical control, because it addresses the intent rather than the symptom.

What a strong answer adds

Treating the monitoring platform as production infrastructure with its own SLO, capacity plan and blast-radius controls. It is the system you depend on when everything else is broken, and it is routinely the least protected component in the estate.