How production systems keep the cost of watching themselves from outgrowing the systems being watched. Reconstructed from the incident record at GitLab and Datadog, the pricing sheets of three clouds, the source and rejected pull requests of Prometheus and OpenTelemetry, and first-person accounts from Zendesk, eBay, Netflix, Zerodha, Wix and Roblox. A reader leaves able to name their telemetry billing atom, choose between the two containment philosophies on purpose, and price the funnel before the finance team does.
A system's description of itself grows faster than the system, and the mature response, everywhere the record is public, is industrial-scale deletion.
Strip the product names away and the problem is this: every request through a modern system emits a description of itself, that description is multiplied by the number of services it touches and the number of dimensions each one records, and the multiplication compounds faster than traffic. Datadog's own storage team measured its data volumes growing 30x between 2017 and 2022. Netflix hit the wall as early as May 2011, when, at roughly two million distinct time series, "the monitoring system was regularly failing to keep up with the volume of data". The organisations that survived this did not buy their way out with bigger storage. They built funnels.
The surprise in this record, and the reason to read on, is how much gets thrown away by the people who take observability most seriously. Google's Dapper kept one trace in 1,024 at generation and then sampled again at the collection tier, "often as low as 0.01% for high-traffic services", and still wrote more than a terabyte of sampled trace data per day in 2010. Zendesk, in a guest post published by its own vendor, reports 1,435 log exclusion filters in steady state, the most common of which drops 99 to 100% of what it matches, and an ingestion-to-indexing ratio of 10:1. Deletion is not a failure of observability discipline. Deletion, done at named stages with named keep-rates, is what observability discipline looks like at scale.
The second thing the record teaches is what the meter actually runs on. For metrics, every pricing sheet and every out-of-memory incident points at the same unit: the series, one unique combination of a metric name and its tag values, alive in RAM somewhere. Datadog bills it by name, AWS prices the metric-month, and GitLab's ingesters OOMed on "high numbers of in-memory series". Volume, gigabytes and datapoints per second, is what logs and traces are priced in; cardinality is what metrics are priced in, on both the invoice and the heap.
Scope: this guide covers cost and scale containment for metrics, logs and traces in production, the mechanisms, the prices and the incidents. It deliberately does not cover alert and dashboard design, compliance-driven retention for security logs, continuous profiling, or feature comparisons between vendors.
This guide was assembled inside a build environment whose network policy allowlists a small set of hosts. Well-known accounts from Uber, Slack, Cloudflare, Pinterest, Shopify, Grafana Labs and Honeycomb's blog could not be fetched, so they are not cited; several papers are cited from byte-identical mirrors with the canonical venue named. The widely reported "$65M per year" Datadog customer from the May 2023 earnings call is deliberately absent for the same reason: none of its primary sources could be verified here, and this guide does not cite what it could not fetch.
Six layers recur across every published system; what varies is where each organisation puts the deletion decision, and who is allowed to make it.
Start at the bottom of the money, not the top of the data flow. The hot tier is a RAM budget, and everything upstream exists to protect it. Facebook's Gorilla stated the constraint plainly in 2015: two billion series at 16 bytes per point would need 16 TB of RAM, so the team compressed points to an average of 1.37 bytes and bounded the window to 26 hours. Google's Monarch is the same shape a generation later, "close to a petabyte of compressed time series data in memory". Grafana's Loki documentation writes the rationale into its per-tenant stream cap: "Active streams are held in memory buffers in the ingesters, and if this value becomes sufficiently large then it will cause the ingesters to run out of memory". When you see a limit knob in any of these systems, read it as a RAM fence with a price tag.
The layer above the store is the edge, and its defining property is that it refuses.
Loki's distributors answer over-limit tenants with named 429s. Prometheus's scrape limits,
sample_limit, label_limit and friends, go further: when one
trips, "the
entire scrape will be treated as failed", nothing partial, nothing silently dropped.
Canopy, Facebook's tracing system, rate-limits with a token bucket "both
globally and per-tenant". Three independent systems, one design conviction: at the
edge, the writer is untrusted, and refusal is the only response that keeps the blast
radius on the writer's side of the line.
The decision layer between edge and store is where organisations diverge, because it is where judgement lives. Dapper put a uniform coin-flip in the SDK and a second, more aggressive sampler at collection. Canopy replaced the coin-flip with named sampling policies, each carrying an owner, a strategy, restraints and a lifetime; policy, not physics. Zendesk's version is 1,435 exclusion filters maintained by hand against a two-price contract. Netflix's version is age-based: hold everything briefly, then "rollup by dropping the node dimension" at four days and again at sixteen. These are four different answers to one question: who signs off on what gets forgotten, and when.
Two boxes in figure 2 are easy to miss and both are documented cost levers. Agent topology: eBay measured a per-node DaemonSet costing roughly 200 MB and 0.6 cores per instance of pure overhead, and moving to cluster-local scraping agents saved "roughly 90%" of that resource bill at a fleet of 1.5 million endpoints. And the query path: GitLab's readiness review for Mimir names the "poison query" as a risk that survived its whole platform migration, and Prometheus now has a design proposal for pre-execution query cost estimates on the grounds that "a single expensive query can hurt a whole Prometheus". Containment that only guards the write path is half a containment.
Samplers, exclusion filters, aggregation and rollup rules. The one layer where every organisation's answer differs, because it encodes who may decide what is forgettable. Canopy gives each policy an owner and a lifetime; Zendesk curates filters by hand; Netflix keys the decision to age.
In-memory series storage with a deliberately bounded window. Gorilla held 26 hours; Atlas holds recent data in memory because query latency demands it; Mimir ingesters hold active series and OOM when churn spikes. RAM here is the physical form of the cardinality bill.
Object storage or columnar files holding what the hot tier aged out, queryable slowly or after rehydration. Datadog built Husky to burst readers over it on demand; Zerodha holds 30 billion log lines in 2.5 TB of ClickHouse; Nexthink archives to S3 without indexing at all.
Three forks carry most of the consequences, and the record contains a genuine argument about each one.
The sharpest recorded argument is about what a limit should do when it trips. In April
2026 a contributor opened a pull request adding a hard cardinality_limit to
the OpenTelemetry Collector's batching layer. Five reviewers approved it. It died anyway,
closed unmerged in September 2026, on one maintainer's objection:
"I'm not sure I understand the purpose of adding this failure point... memory is bounded without rejecting anything. If we want to add this hard cap for some reason, we should remove LRU." dmitryax, OpenTelemetry Collector pull request 15225, September 2026
That sentence names the two containment philosophies cleanly. Rejection fails loudly at
the boundary: Prometheus fails the whole scrape, Loki returns a 429 with a named reason,
and the operator finds out immediately. Eviction degrades quietly inside: the OTel SDK
spec caps cardinality by folding excess series into a single
otel.metric.overflow series, an LRU keeps memory bounded, and nothing pages.
The collector maintainers judged that stacking a rejection knob on top of an eviction
design adds a failure point without adding safety. Prometheus, from the opposite pole,
has kept a maintainer's own
proposal for scrape-time cardinality capping open and unresolved since 2022, partly
because a partial drop would blur exactly the loud-failure semantics it relies on.
The second fork is where the trace keep-decision runs. Head sampling decides at the root span, cheaply: Dapper's uniform 1/1024, refined by adaptive sampling parameterised by desired traces per unit time. Tail sampling holds every span of a trace in memory until it can judge the whole thing, which is how you keep every error and every slow outlier: Honeycomb's Refinery keeps "one out of every 1,000 traces for requests that return 2xx" while keeping every 5xx. The price is written in the OTel tail sampler's own README: the buffer is a circular one, and "this can cause a trace to be dropped before it's sampled", while every knob that reduces that risk increases memory. A 2025 issue names the consequence for the bill: "memory usage scaling linearly to throughput. Higher memory requirements translates to higher hardware cost." OTEP 235 exists precisely because real systems end up doing both, and two samplers that decide inconsistently destroy the statistics; its threshold mechanism makes head and tail stages compose.
The third fork is commercial: drop before ingest, or ingest everything and index a fraction. The vendors have built a two-price split for the second option, and the practitioner accounts show both being worked hard. Zendesk ingests ten times what it indexes and holds the line with exclusion filters. Nexthink goes further on the cheap side, archiving logs directly to S3 "without ingesting or indexing them". Zerodha opted out of the split entirely, moving logs from ELK to self-hosted ClickHouse: 30 billion log lines in 2.5 TB where the old stack needed 13 TB for the same order of volume, at less than half the hourly infrastructure cost. The flip condition is not subtle: the two-price split is worth paying for when someone else runs the platform and your query patterns are unpredictable; it stops being worth paying for when your ingest is dominated by data nobody has ever queried, which is exactly what Zendesk's 99-to-100% filters say out loud.
| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Limit semantics | Fail the whole scrape (Prometheus) | Partial silent drop | A visible refusal beats an unnoticed gap | Trusted single-team pipeline: evict to overflow and alert | Prometheus docs |
| Cap in collector batching | LRU eviction only | Hard cardinality cap (5 approvals) | "Memory is bounded without rejecting anything" | The cap is the product being sold, per-tenant, at a platform edge | PR 15225 |
| Trace keep-decision | Head at 1/1024 (Google) | Full tracing | 16.3% measured latency at 1/1 | Errors are the point and RAM is affordable: go tail | Dapper |
| Log economics | Ingest 10x, index 1x (Zendesk) | Index everything | Most matched volume is never queried | Ingest itself dominates: drop at source or self-host columnar | Zendesk, Zerodha |
| Metrics retention | Age-tiered rollups (Netflix) | Uniform retention | Old data is queried at coarser grain | Regulated workloads that must replay raw history | Atlas docs |
| Buy or self-host metrics | Self-host (Zerodha, Wix, Grammarly) | SaaS per-series pricing | Reported savings up to 10x at high churn | No team to own a new incident class; GitLab's inter-AZ network bill shows self-hosting has its own meter | Case studies, GitLab |
| Agent topology | Cluster-local scraping (eBay) | Per-node DaemonSet | Roughly 90% resource savings at 1.5M endpoints | Node-level signals (kernel, disk) that only a node agent can see | eBay |
The incidents cluster into three classes: the watcher eats itself, the watcher fails closed, and the containment machinery degrades the watching silently.
Class one is self-inflicted cardinality, and it is the best documented because GitLab publishes its incident reviews in a public tracker. The pattern is always the same mechanism at different magnitudes: something multiplies label values, the in-memory series count climbs, and the process that holds them dies. At GitLab in October 2025 the multiplier was routine pod churn from frequent deployments; 18 of 210 Mimir ingesters OOMed, quorum broke, and metrics writes failed platform-wide for 25 minutes. At token.place in September 2026 the multiplier was default Flask instrumentation keeping request-controlled URL paths as label values; the series count went from 28,964 to 71,056 in eighteen minutes and the scrape payload alone reached 7.79 MB every 30 seconds. Same curve, five orders of magnitude apart in fleet size. The transferable rule: any label whose values are chosen by traffic rather than by an engineer is an unbounded meter, and the meter runs whether or not anyone is watching it.
One class is conspicuously missing from the public record: the bill as the incident. Practitioner postmortems for observability cost overruns, written by the team that paid, are rare to the point of absence; the genre is dominated by competing vendors' marketing. GitLab's cost tracker is the nearest thing, and its finding is instructive: the surprise line item in self-hosted metrics was "much higher than expected inter-zone AZ traffic costs" from replication, not storage or compute, and the levers it considered, single-AZ operation and spot instances, trade away exactly the redundancy the platform exists to provide. Treat this gap as information: if you want to know what a cost blow-up looks like before the invoice, nobody has published the timeline, so instrument your own meter (rung 1 of the ladder) rather than waiting to read about someone else's.
Scale points from the systems that published them, and list prices from the three meters an architect is most likely to face.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Active series | 3 billion | eBay | Sherlock.io, 1.5M scraped endpoints, 40M samples/s | 2022 | eBay on OTel blog |
| Active series | 5 billion | Roblox | Self-reported, 120M datapoints/s ingest | 2023, checked 2026 | VM case studies |
| Series churn | 150M new/day | Wix | Against 50M active per instance; churn, not active count, drives index growth | checked 2026 | VM case studies |
| Hot-tier compression | 1.37 bytes/point | Down from 16; made 26h in RAM affordable at 12M points/s | 2015 | Gorilla, VLDB | |
| In-memory store | ~1 PB | Monarch, compressed, terabytes/s ingest, 38 zones | 2020 | Monarch, VLDB | |
| Trace sampling | 1/1024 head, to 0.01% at collection | Full tracing measured at 16.3% latency overhead; sampled residue still >1 TB/day | 2010 | Dapper | |
| Traces recorded | 1.3 billion/day | Canopy, per-tenant token buckets at the edge | 2017 | Canopy, SOSP | |
| Single-server bound | tens of millions of series | DigitalOcean | 192 Prometheus servers, 200M+ series, 2M+ samples/s fleet-wide | 2018 | KubeCon slides |
| Log storage ratio | 2.5 TB vs 13 TB | Zerodha | 30B lines in ClickHouse vs 28B in ELK; node cost $1.14/h vs $2.67/h | 2023 | Zerodha |
| Ingest-to-index ratio | 10:1 | Zendesk | 1,435 exclusion filters; 4x volume cut on the hottest service | 2025 | Zendesk |
| Custom metrics price | $5 / 100 / month indexed; $0.10 / 100 ingested | Datadog | A metric is one name-and-tag-value combination; histograms multiply it | 2026 list | Pricing list |
| Metric-month price | $0.30 tiering to $0.02 | AWS CloudWatch | First 10k at $0.30, past 1M at $0.02; logs $0.50/GB ingest, $0.03/GB-mo storage | 2026-09-22 | Price List API |
| Log ingest price | $0.50/GiB incl. 30d | Google Cloud | Then $0.01/GiB-mo; monitoring $0.2580/MiB tiering down; managed Prometheus $0.06/M samples | 2026, logging price effective 2024-10 | GCP pricing |
| Indexed log events | $1.70/million (15-day) | Datadog | Versus $0.05/million in Flex storage: a 34x spread inside one product | 2026 list | Pricing list |
| Worked mid-market bill | ~$18,400/month | OneUptime scenario | Competitor's list-price model for a 50-person team; a claim, not a measurement | 2026-03 | OneUptime |
A workable cost model needs only three variables, one per signal. Metrics: active series times price per series-month, where the series count is driven by churn (Wix's 150 million new series a day against 50 million active is the cleanest published illustration that churn dominates). At Datadog's list price of $5 per 100 per month, an application emitting 10,000 custom series costs $500 a month before a single log line; the same 10,000 on CloudWatch's first tier is $3,000, falling steeply with volume discounts. Logs: gigabytes ingested times the ingest price, plus events indexed times the index price; Datadog's own sheet spreads 34x between an indexed event and a Flex-stored one, which is the entire economic argument for the Zendesk pattern. Traces: spans kept per second times retention, where the keep-rate is your sampling policy; Dapper's numbers say the runtime overhead argument ends at roughly 1/16 and everything below that is storage economics. These are derived illustrations from the linked list prices, not quotes; negotiated contracts sit below list, sometimes far below.
Scale figures from eBay, Roblox, Wix, Zerodha and Zendesk are self-reported and unaudited; the Roblox and Wix rows live in a vendor-curated document. Grafana's "1 billion active series" for Mimir is the vendor's own internal test, labelled as such. All prices are 2026 list prices for US regions and go stale silently; the CloudWatch row carries its exact publication date because the Price List API stamps one. No independent measurement exists for any SaaS-versus-self-host comparison in this corpus; every migration figure comes from the team that chose to migrate.
Every source behind this page, graded. The postmortems and the rejected pull request are the load-bearing items; read those two tiers first.
Pod churn drove in-memory series high enough to OOM 18 of 210 ingesters; quorum loss turned 8.5% component failure into a 100% write outage for 25 minutes.
A systemd auto-update deleted CNI routes fleet-wide; five regions, all products, customers' monitors "unavailable and not alerting"; 500 to 750 engineers in shifts.
Names the false assumption behind the 2023 outage in one phrase: a bias toward "full stop over showing almost correct data", producing a "square-wave failure pattern".
The recording rules that watch the metrics platform generated the high-cardinality query load that destabilised it; fixed by raising the meta-tenant's querier quota.
Default Flask instrumentation kept request-controlled paths as labels: 28,964 series to 71,056 in eighteen minutes, 7.79 MB scrape payloads, 87 minutes of outage. A small system, which is why the whole mechanism is visible end to end.
Ten hours of degraded visibility in which collection was healthy and the query layer was not; detection was a human noticing Grafana timeouts.
Thanos rule evaluators OOMed; apdex and other derived metrics vanished for nearly two hours while raw ingestion stayed healthy. Rated HIGH with zero customer impact.
An OOM crash loop from WAL corruption, worked as a production incident with corrective actions; 45 minutes to mitigate by deleting the corrupt segment.
A hard cap with five reviewer approvals, rejected on one maintainer's argument that eviction already bounds memory and a rejection knob adds a failure point. The clearest recorded statement of the two containment philosophies.
num_traces (50,000 default), decision_wait (30s default), and a circular buffer whose documented behaviour is that a trace can be dropped before its keep-decision runs.
Every scrape limit defaults to off, and each one that trips fails the entire scrape. The all-or-nothing semantics are a deliberate design position, not an accident.
A maintainer's own proposal to cap unique label values per scrape, open for four years across 15 comments. Upstream has no agreed answer to scrape-time cardinality capping.
Operators report that hitting the limiter "could lead to full vmstorage degradation since it goes via the slowest path", and that the hourly/daily dual semantics confuse the people who must configure them.
The SDK-level eviction answer: past the cardinality limit, excess series fold into a single otel.metric.overflow series. Stabilised only after Go, C#, Java and Rust had all shipped it.
"When sampling is to be performed at multiple points in the process, the only way to reason about it effectively is to make sure that the sampling decisions are consistent." The mechanism: a rejection threshold carried in TraceState, non-decreasing along the path.
Not yet implemented: pre-execution cost estimates and hot-reloadable per-query ceilings, motivated by "a single expensive query can hurt a whole Prometheus" and the fact that today's limits require a restart to change.
Operational design doc for volume containment as layered per-tenant refusals, with the memory rationale for each limit written next to its default and its recommended ceiling.
The migration record: 18 months of Thanos-related incidents drove the change, the new system ran side by side from November 2023, and the review names the poison-query risk that migration does not remove.
The steady state of a marquee SaaS observability customer: 1,435 exclusion filters, the most common excluding 99 to 100%, a 10:1 ingest-to-index ratio, and a 4x volume cut on the hottest service.
Sherlock.io at 1.5M scraped endpoints, 40M samples/s, 3B active series; moving from per-node DaemonSets to cluster-local agents saved roughly 90% of the agent resource bill.
The 2011 drowning at two million series, and the containment that followed: full resolution briefly, node dimension dropped at four days, all metrics rolled up at sixteen, whitelist beyond.
ELK to ClickHouse: 30 billion lines in 2.5 TB against 13 TB before, node cost less than half, retention tiered by log level. A complete self-host counter-example to the two-price split.
30x data growth in five years forced an index rewrite; the rewrite bought 20x higher queryable cardinality on identical hardware and halved the indexing service's cost.
The event store behind the two-price split: storage decoupled from compute so a year-deep query can burst readers over cheap storage on demand.
First-person scale figures from teams that left Prometheus-family or SaaS stacks: Wix at 50M active series with 150M/day churn, Roblox at 5B active series and 120M datapoints/s, Grammarly reporting 10x cost reduction. Vendor-curated but quoted first-person.
Archive to S3 without ingesting or indexing, keep custom metrics at 9.9k against 141 integrations' worth of included metrics, "for a similar budget, we are now processing significantly more data".
1/1024 head sampling, a second collection-tier sampler down to 0.01%, full tracing measured at 16.3% latency overhead, and still more than a terabyte of sampled traces a day. Fetched from Google's own publication bucket.
16 bytes per point to 1.37 via XOR compression, a 26-hour in-memory window as an explicit requirement, and clusters that doubled twice under growth anyway.
The ceiling of the arms race: close to a petabyte of compressed series in RAM, terabytes per second of ingest, 38 zones. What "just keep everything" actually costs at the limit.
1.3 billion traces a day contained by edge sampling policies that carry an owner, a strategy, restraints and a lifetime, backed by global and per-tenant token buckets.
Slide 11 gives the log cost equation; slide 12 draws Dapper's retention funnel, 100% to 0.01%; slide 24 prices cardinality as dollars per tag value.
192 Prometheus servers, 200M+ series, 2M+ samples/s (slide 24), and the operational bound: queries should touch hundreds, not thousands, of series (slide 26).
Practitioner root-cause list for cardinality cost (user_id, request_id, url, ip as labels, slide 10) and an over-collection worked example: "you're over-collecting by 12x" (slide 11).
The billing atom in writing: a custom metric is one name-and-tag-value combination including the host tag; $5 per 100 indexed per month against $0.10 per 100 ingested, and a 34x spread between indexed and Flex-stored log events.
Metric-months tiering $0.30 down to $0.02, logs at $0.50/GB ingest and $0.03/GB-month storage, pulled from the machine-readable price file with its own publication date.
Logging $0.50/GiB with 30 days included then $0.01/GiB-month; monitoring by ingested MiB tiering from $0.2580; managed Prometheus by the million samples, $0.06 tiering to $0.024.
Mimir's "1 billion active time series" is internal testing; Refinery's README documents dynamic sampling mechanics; OneUptime's ~$18,400/month worked bill is a competitor's list-price model. Each used here as a labelled claim, not a measurement.
Seven rungs from an evening's experiment to a priced, bounded pipeline. The crossing from toy to real happens at rung four.
Point promtool tsdb analyze (or your store's cardinality endpoint) at an existing setup and rank the top ten metrics by series count and the top ten label keys by distinct values.
Done when: you can name the three metrics that dominate your series count and who owns them. Teaches: cardinality is measurable before it is billable.
In a sandbox Prometheus, add a label carrying a request ID to one metric and watch series count and memory climb; then set sample_limit on the scrape and watch the whole scrape fail instead.
Done when: you have seen both outcomes and the alert that fires (or fails to fire) for each. Teaches: rejection semantics, and why the failed scrape is the honest failure mode.
Configure an OTel SDK cardinality limit on the same workload and find the otel.metric.overflow series in the output; alert on its presence.
Done when: the overflow series appears under load and pages you. Teaches: eviction semantics, and that the overflow series is containment's only witness.
Stand up an OTel Collector with a tail sampler biased to errors: keep every 5xx trace and roughly 1% of 2xx. Load it until the circular buffer starts dropping undecided traces, and record the memory-per-throughput slope while it does.
Done when: error traces survive a load test at 100% while 2xx traces sit near the target rate, and you know the RAM cost per thousand traces per second. Teaches: the decision layer's economics, on the exact knobs the README warns about.
Route the same log stream to an indexed store and, in parallel, to object storage or ClickHouse. Measure the per-gigabyte cost of each path and time a query against each tier.
Done when: you can state your own indexed-versus-archived cost spread and the query-latency price of the cheap tier. Teaches: the two-price split from the inside.
Run Loki or Mimir with per-tenant limits, drive one tenant past its stream or series cap, and verify the 429s land on that tenant while the neighbours stay healthy.
Done when: one tenant's overload is that tenant's problem, provably. Teaches: blast-radius containment at the edge, the property every platform team is actually buying.
Take the series counts, gigabytes and span rates from rungs one to six and compute the monthly bill under Datadog, CloudWatch and GCP list prices; then compute the self-host cost including the inter-AZ replication traffic GitLab found dominating.
Done when: a one-page unit-economics sheet names your billing atom per signal and the utilisation at which each option flips. Teaches: the argument you will actually have in the design review.
The queries that found this material, grouped by what they surface. The last group is the workaround worth keeping: when a blog host is unreachable or dead, its static-site source repository often is not.
"active time series" "churn rate""exclusion filter" logs cost "we""samples per second" "active series" migrationCaseStudies.md site:github.com metrics"cardinality explosion" postmortem OOMsite:gitlab.com gl-infra "incident review" mimir"in-memory series" OOM ingester quorum"monitors were unavailable" outage telemetryrepo:open-telemetry/opentelemetry-collector is:pr is:closed is:unmerged cardinality"entire scrape will be treated as failed"repo:prometheus/prometheus is:issue cardinality limit scrapeotel.metric.overflow stabilizepricing.us-east-1.amazonaws.com offers AmazonCloudWatch index.json"per metric-month" OR "per 100 custom metrics"site:raw.githubusercontent.com <blog-name> content blog<paper title> filetype:pdf site:raw.githubusercontent.com