A payments team pages on `increase(payout_failures_total[5m]) > 2`. That counter moves about three times an hour. The page fired twice on a quiet Sunday and stayed silent through nine failures spread over 40 minutes last Tuesday. Work out what the expression actually returns and what you would alert on instead.

The assumptions, stated

Three failures an hour is 0.05 per minute, so the expected count inside a 5-minute window is 0.25. The threshold is > 2, which is eight times the expected value. Scrape interval 30 seconds, rule evaluation every 30 seconds.

The arithmetic the query actually does

increase() is documented to extrapolate its result to cover the full range in the selector, which is why it can return a non-integer even though the counter only moves in integer steps. The samples inside a 5-minute window span at most 4.5 minutes of real observations, so a single increment that lands between the first and last sample is scaled up: one event observed across 4 minutes of a 5-minute range reports roughly 1.25, and two events report roughly 2.5.

That is the Sunday page. Two failures crossed a threshold whose author meant "more than two", because extrapolation inflated the reading by about 25%. The same mechanism explains the silence on Tuesday: nine failures over 40 minutes is 1.1 per 5-minute window, extrapolated to maybe 1.4, and the alert never gets near 2 even though the failure rate rose roughly fourfold.

Which assumption dominates the error

Not the extrapolation factor, which is bounded by the ratio of the range to the observed span. It is the window length against the event rate. With an expected count of 0.25, the observed count is dominated by arrival noise: the window either contains an event or it does not, and no threshold placed on that quantity distinguishes a 4x rate change from an ordinary Sunday. A short window on a sparse counter has no statistical power, and tuning the threshold cannot create any.

Two related rules fall out of the same arithmetic. A range needs at least two samples before a rate can be computed at all, so a 1-minute window on a 30-second scrape is one missed scrape away from returning nothing; four scrape intervals is the usual floor. And rate() must be applied before any aggregation, because a sum() taken first hides the per-series counter resets that a restart produces.

What to alert on instead

• Alert from the event, not the aggregate. At three per hour a human can read every failed payout, with its reason and its correlation ID, deduplicated by reason. Choose the event record when the expected count per window is under five; the metric was never the right instrument for a signal this rare. It costs a log-pipeline rule and a deduplication key rather than a metric rule.
• Keep the counter for comparison over a long window. increase(...[6h]) has an expected count of 18, where a few per cent of extrapolation error is irrelevant, and comparing it to the same window a week earlier catches the slow drift that the 5-minute rule cannot see.
• Require a denominator. If a ratio alert survives, gate it on a minimum volume in the window so a quiet period cannot fire it.

When this is the wrong answer

On a counter doing thousands of increments per window in production, the extrapolation is a rounding error, rate() over 5 minutes is correct, and event-level alerting would be a self-inflicted flood. The fix is scoped to counters whose expected count per window is in single digits. The test is arithmetic, not taste: compute the expected count in the window before writing the threshold.

Common weak answers

• "Lengthen the window to an hour." It fixes the power problem and makes detection an hour late. Pair a long window for trend with an event-level signal for the ones that matter.
• "Use a gauge of current failures." A gauge cannot be rated or summed correctly and loses every event between scrapes.
• "Add for: 5m." That suppresses flapping. It does not repair a threshold set at eight times the expected value.