Counter Reset Handling

A counter in a metrics system is cumulative: it only ever increases, for the lifetime of the process that owns it. When the process restarts the counter starts again at zero, which would produce a large negative delta and an absurd rate.

Every rate function therefore encodes one rule: if the value went down, assume a reset, and add the last value seen before the drop into the delta. That single convention is why a rate() graph looks continuous across a deploy that replaced every instance, and it is the reason counters are the cheapest correct way to measure throughput in a fleet that is constantly being replaced.

It is also a convention with two sharp edges, and both fail quietly in production.

Why it matters

It decides whether a throughput graph is a measurement or an artefact. Teams reach for a counter because the name fits, not because the value is monotonic, and the system will not complain. A queue depth, a connection count or an in-flight gauge exported as a counter produces a graph that moves whenever the value dips, which looks like traffic and is not.

The second edge matters for architecture rather than instrumentation: increments made by a process that lived less than one collection interval are never observed at all. With a collection interval of 15 seconds, a pod that lives 8 seconds contributes nothing, which quietly rules out scrape-based counters for short-lived work.

Implementation patterns

• Declare the type honestly. Counter for things that only go up (requests served, bytes written, errors); gauge for anything that can fall; histogram when you need the distribution.
• Never reuse a series name across a type change. Changing a metric from gauge to counter in place leaves a series whose history is half one semantic and half the other, and every query over a window spanning the change is wrong.
• For jobs shorter than the collection interval, push the final value to an aggregating sink or emit the work as events. A 15-second scrape cannot see a pod that lived eight seconds.
• Reset on restart, not on a business event. Zeroing a counter at midnight, or when a tenant is reset, manufactures a reset the query engine will faithfully compensate for, inflating the rate by the whole pre-zero value.
• Prefer increase() over a window that comfortably exceeds the scrape interval, because extrapolation at window edges is the other place small errors enter.

Industry example

The convention is implemented in Prometheus and inherited by the OpenTelemetry metrics model, whose specification distinguishes monotonic sums from non-monotonic ones precisely so that a backend knows whether a decrease means a reset or a real decline. The fact that two ecosystems independently made this a type-level distinction rather than a query-time heuristic is the clearest evidence of how often it is got wrong.

Failure scenarios

• A gauge exported as a counter: every dip is read as a restart, the rate is inflated by the size of the dip, and the graph correlates with traffic patterns convincingly enough to pass review.
• Two restarts inside one interval: only one decrease is observed, so one restart's worth of increments is lost.
• Serverless or batch work under a scrape model: most increments are invisible, and the graph understates throughput by an amount nobody can calculate.
• A counter reset by application logic at a rollover boundary, producing a spike in every rate query at the same time every day.
• Counter resets during a rolling deploy being read as a traffic drop by an autoscaler that scales in at the worst moment.

Trade-offs

Cumulative counters buy robustness: a lost scrape costs resolution but not correctness, because the next sample still carries the total. Delta-reporting systems lose data permanently when a report is lost, but they handle short-lived processes naturally. You are choosing between tolerating lost samples and tolerating unobserved processes, and which one hurts depends on whether your workload is long-lived or ephemeral.

When not to use it

Do not model anything that can legitimately decrease as a counter, however tempting the aggregation is. Queue length, active connections, cache entries and temperature are gauges; their rate of change is a derivative you compute explicitly and interpret knowing that negative values are real. And for anything you will invoice from, prefer an acknowledged event stream over a scraped counter — a counter whose samples can be missed is a fine operational signal and a poor ledger.

Interview question

Q: A team exports queue depth as a counter and alerts on its rate. The alert fires several times a day and the queue is fine. Explain the mechanism, and tell me what you would change and what you would check for elsewhere in the estate.

What a strong answer covers: that every decrease is compensated as a reset, so the rate is inflated by the size of each dip; that the graph looks plausible, which is why it survived; the fix (declare it a gauge, do not reuse the series name); and the audit — search the metric registry for counters whose values ever decrease, which finds the rest of them in one query.

Quick check

Quiz: Why does a rate graph stay continuous across a deploy that replaces every instance? Because a decrease is interpreted as a reset and the pre-reset value is added back into the delta, so at most one interval's increments are approximated.

Flashcard: When does counter-reset handling silently lie? When the series is really a gauge, so ordinary dips are treated as restarts and inflate the rate; and when a process lives less than one collection interval, so its increments are never observed at all.