Metrics
Cheap pre-aggregated numeric time series — excellent for knowing something is wrong, structurally unable to tell you which request.
Definition
A metric is a numeric measurement aggregated over time and dimensions. Counters, gauges, histograms. Cost is roughly independent of traffic volume and roughly proportional to the number of distinct label combinations.
What metrics are for, and what they are not for
For: detecting that something is wrong, alerting on it, capacity trends, SLO measurement, comparing now with last week. Cheap, fast, retained for a long time.
Not for: explaining why, or finding the specific request that failed. A metric has already thrown away the individuality of every event that produced it. That is the trade that makes it cheap, and no amount of dimensionality recovers it.
The correct mental model: metrics tell you to look, traces and logs tell you where.
Cardinality, the mistake everyone makes once
Cost and storage scale with the number of unique label combinations. Adding user_id as a label to a
metric with a million users creates a million time series, and it will either bankrupt the
observability budget or take down the metrics backend.
The rule: labels must be bounded and low-cardinality — service, endpoint, status class, region. Anything unbounded — user ID, request ID, URL with parameters, error message text — belongs in a log or a trace, never in a metric label.
The four that matter most
The RED method for request-driven services — Rate, Errors, Duration — plus saturation of the constrained resource. The USE method for resources — Utilisation, Saturation, Errors. Between them they cover most of what a service needs, and a team with these four in place is ahead of most.
Always percentiles, never averages. An average latency of 200 ms is compatible with 90% of requests at 50 ms and 10% at 1.5 s. The average describes nobody's experience; the p99 describes the worst of it.
A subtlety worth knowing: percentiles do not average across instances. The mean of ten instances' p99 values is not the fleet p99. Aggregate the underlying histograms, not the computed percentiles.
Failure scenarios
- High-cardinality labels taking down the metrics system.
- Averages hiding the tail, so a dashboard is green while a tenth of users suffer.
- Metrics that measure the system rather than the user. CPU is fine and checkout is broken.
- Alerting on a metric with no corresponding user impact, which produces pages nobody can act on.
- Percentiles averaged across instances, producing a number that is not a percentile of anything.
Interview question
"Why can a dashboard show every service healthy while users cannot complete a purchase?"