A design platform of the kind Canva runs exports 40 metrics per service. An engineer adds a `pod_name` label so a noisy pod can be identified. The service runs 600 pods and deploys twice a day, so pod names turn over completely every 12 hours. Retention is 30 days. Roughly how many distinct series does that one label create over the retention window?
Show the full answer Hide the answer
The assumptions, stated
- 40 metrics exported per service instance.
- 600 pods live at any moment.
- Two deploys a day, each replacing every pod, so pod names are completely new twice a day.
- 30-day retention, and the store indexes a series for as long as any of its samples are retained.
The arithmetic
40 metrics × 600 pods = 24,000 series per pod generation. That is the number alive at any instant, and it is the number most people answer with.
Two generations a day × 30 days = 60 generations, each with entirely new label values. 24,000 × 60 = 1,440,000 distinct series that the index must hold for the window.
The correct mental model is that cost follows churn, not concurrency. A series is created the first time a label combination appears and must be findable until its last sample ages out. A label whose values are stable costs what you see today; a label whose values are regenerated on every deploy costs that multiplied by the number of deploys in the retention window.
Which assumption dominates the error
Deploy frequency, by a wide margin. Move to continuous deployment at ten releases a day and the same label produces roughly 7.2 million series over the window — a 5x swing from a number nobody thinks of as a monitoring decision. Pod count and metric count enter linearly and are known; deploy frequency is the one that changes without anyone revisiting the label.
Why the other options fail
- 24,000 is the instantaneous active series count. It is the right answer to a different question and the reason this mistake is so common: the dashboard showing active series looks fine while the bill and the index grow.
- 240,000 assumes churn over ten days, or one deploy a day for ten days — the right method with the wrong window.
- 24 million overshoots by treating every pod as unique per scrape interval rather than per deploy, which would be the answer if the label were something genuinely per-request.
What the number rules out
At roughly 1.4 million series for one label on one service, this does not generalise across an estate. Twenty services with the same well-intentioned label is 29 million series, which is the scale at which query latency degrades and ingestion starts dropping.
When the label is worth it and when not
Keep pod identity on two or three diagnostic metrics — restart count, memory working set, request errors — and leave it off the other 37. Better still, attach the pod identity to an exemplar on the histogram bucket, or to a wide structured event, so you can jump from "the tail moved" to a specific pod without every metric carrying every pod name forever. The rule: identity that answers "which one" belongs on events and exemplars; labels are for dimensions you group by.