intermediate 3 min answer

Two requests arrive the same week. Team A wants its latency metric scraped every 5 seconds instead of every 30 so its burn-rate alert reacts faster. Team B wants one extra label with 40 values on the same metric. The metrics bill is already under scrutiny. Which request costs more, and what do you give each team?

costcardinalityresolutionprometheusexemplars
Show the full answer Hide the answer

What is gained and what is paid

The two requests look comparable and are not, because samples compress inside a series and series do not compress at all. Prometheus documents an average of only 1 to 2 bytes per sample, and states directly that reducing the number of series is the more effective lever because samples within a series compress well.

Team A, in numbers. One series at a 30-second interval is 2880 samples a day, roughly 3 to 6 KB. At 5 seconds it is 17280 samples, roughly 17 to 35 KB. Six times the samples, the same number of series, an unchanged index, and six times the ingest and write-ahead-log work. Linear and recoverable: change the interval back and the cost stops.

Team B, in numbers. A histogram with 12 buckets plus its sum and count is 14 series per label combination. If the metric already carries 8 routes by 3 status classes, that is 336 series. One label with 40 values makes it 13440. The new label multiplies both the stored samples and the index, and index size and per-series memory are what decide whether the metrics tier stays up. Multiplicative and sticky: the series live out their retention whatever you do next.

So B is roughly 40 times on the dimension that is hardest to undo, and A is 6 times on the cheapest one.

What to give each team

  • Team A gets 10 seconds on its indicator metrics only, and a better answer to its real question. A burn-rate alert's detection latency is dominated by the evaluation window and the hold-down clause, so 5-second scraping underneath a 5-minute window buys nothing anyone can act on. The argument worth having is about shortening the window.
  • Team B gets the label refused and two substitutes. Exemplars attach a trace ID to a histogram bucket and add no series at all, which answers "show me one of these" exactly. A wide-event stream answers per-value questions at query time, where a 40-value dimension and a 40000-value dimension cost the same.

When the trade flips

If the tier has series headroom and the label is genuinely bounded and used weekly by on-call, 13440 series may be affordable. Compute the number, then decide; the rule is not "no labels" but that a label is paid for in series and must be bounded where it is created, with an ingestion limit that fails loudly rather than arriving as an invoice.

Higher resolution is the right spend when a machine consumes the metric. An autoscaler input, a load-shedding trigger or a traffic-shift controller deserves 1 to 10 seconds, because the control loop's period sets the requirement rather than a human's attention span.

Common weak answers

  • "Both are small, approve both." One is additive and the other multiplicative. Approving multiplicative requests one at a time is the standard way a metrics tier dies.
  • "Cut retention to pay for it." Retention is the cheapest component of a series and the first thing an investigation reaches for.
  • "Make it a gauge instead of a histogram." That removes 13 series per combination and the ability to compute a quantile, which is the reason the metric exists.