Scrape Sample Limit

A release adds a label carrying a request identifier to one latency histogram. Nothing breaks in the service. By the afternoon the shared metrics tier is out of memory, dashboards for 200 unrelated services time out, and the team that caused it is the last to find out. Removing the label stops the growth and does not undo it: the series already exist and keep answering queries until their retention expires.

The cap exists because cardinality damage is shared, delayed and hard to reverse. The only control that acts before the cost is incurred is a limit enforced at the point of collection, which turns a slow invoice into a fast error with an owner.

Why it matters

Cardinality is the one telemetry cost that is multiplicative rather than additive. A label with 40 values multiplies every existing combination of that metric by 40, and a 12-bucket histogram carries 14 series per combination, so a single commit can add five figures of series. Samples compress inside a series at an average of 1 to 2 bytes each, as Prometheus documents, while each new series carries index entries and per-series memory that do not compress at all.

Enforcement also fixes the incentive. Without a limit, the engineer who adds a label experiences no cost and the platform team experiences all of it, which is why cleanup campaigns decay within two quarters.

Implementation patterns

• Set sample_limit per scrape job, not globally. It defaults to 0, meaning no limit, so it is off until someone chooses a number. Start at two to three times the samples a target currently exposes, which absorbs normal growth and catches an explosion.
• Pair it with label_limit, which catches the other shape of the same mistake: a wide record rather than a numerous one.
• Stage it. First alert when scrape_samples_scraped for a job passes 60% of the intended limit, let teams see their own number for a few weeks, then enforce. Enforcing first produces an incident and a political argument instead of a fix.
• Keep a surgical alternative ready. A metric relabel rule that drops the offending label restores service in one configuration change without failing the whole target, which is what you want during an incident.
• Attribute the limit to a team and publish consumption, so the error has a named owner.

Industry example

The mechanism is documented behaviour that any Prometheus-compatible stack in production inherits: a scrape exceeding the configured sample_limit after metric relabeling is treated as failed in its entirety. Managed vendors implement the same idea at the API boundary, rejecting writes once a per-account series quota is reached. Platform teams usually arrive here from the other direction, after one unbounded label makes querying unusable for everyone.

Failure scenarios

• All-or-nothing loss. Exceeding the limit fails the entire scrape, so every metric for that instance disappears, including the unrelated ones the on-call engineer needs. A limit set too tightly turns a harmless new metric into total blindness for a service.
• Nobody watches scrape failures. The target reports up == 0 and scrape_samples_scraped goes silent, which is detectable and frequently undetected, so the loss surfaces during the next incident rather than at the moment of enforcement.
• Displacement. Teams meeting the limit sometimes delete useful metrics rather than fix the label, so the cap needs to come with the alternative rather than on its own.

Trade-offs

When not to use it

A single team running its own metrics tier has a fast enough feedback loop without it: the person who breaks it is the person who fixes it, the same afternoon. Leave it off, too, for the small set of targets whose loss would be unacceptable — control-plane and infrastructure exporters — and protect those with relabel rules and a churn alert instead, because for them a failed scrape is worse than an expensive one.

Interview question

Q: You own a shared metrics platform for 300 services. How would you stop cardinality explosions without becoming the team that says no to every label?

What a strong answer covers: enforcement at collection with a per-job limit and staged rollout; the arithmetic that makes cardinality multiplicative while resolution is additive; publishing per-team consumption so the cost is visible to its creator; offering exemplars and a wide-event stream as the answer to per-value questions; and the honest cost of the mechanism, which is that a tripped limit removes every metric for that target.

Quick check

Quiz: A target exceeds its configured sample limit. Which metrics are lost? — All of them for that instance: the scrape is treated as failed in its entirety, not trimmed to the limit.

Flashcard: Why enforce a sample cap at collection rather than clean up afterwards? — Because series already created keep costing money and query time until their retention expires, and the creator of the label never feels the cost.