The meter runs on cardinality
How production systems keep the cost of their own telemetry (metric cardinality, log volume, trace sampling) from outgrowing the systems being watched.
Reconstructs the telemetry cost-containment funnel from the incident record at GitLab and Datadog, the pricing sheets of three clouds, Prometheus and OpenTelemetry source debates including a rejected pull request, and first-person accounts from Zendesk, eBay, Netflix, Zerodha, Wix and Roblox. A reader leaves able to name the billing atom for each signal, choose between rejection and eviction as a containment philosophy on purpose, and price the funnel before finance does.
The organisations that take observability most seriously throw almost everything away on purpose: Google's Dapper kept 1 trace in 1,024 and then sampled to 0.01% at collection, and Zendesk runs 1,435 log exclusion filters whose most common setting drops 99 to 100% of what it matches, published on the vendor's own blog.
What you get out of it
- The billing atom for metrics is the series (one name-and-tag-value combination), on the invoice and on the heap alike: Datadog prices it at $5 per 100 per month, AWS prices the metric-month, and GitLab's ingesters OOM on in-memory series counts.
- Containment splits into two philosophies, rejection (Prometheus fails the whole scrape, Loki 429s the tenant) and eviction (OTel folds excess into an overflow series); the OTel collector rejected a hard cardinality cap that had five approvals because stacking both adds a failure point.
- Every mature pipeline is a three-to-four-order-of-magnitude deletion funnel with named keep-rates per stage; Zendesk's steady state indexes a tenth of what it ingests.
- Cardinality bursts come from churn, not steady state: GitLab's Mimir outage was manufactured by routine pod rotation, and Wix reports 150M new series per day against 50M active.
- Self-hosting moves the meter rather than removing it: GitLab found inter-zone replication traffic, not storage, the dominant surprise cost of running its own metrics platform.
Scope
Why this, now. Observability spend has become a board-level line item while the 2025-2026 incident record (GitLab's Mimir cardinality outages, the OTel collector's rejected cardinality cap) shows the containment mechanisms themselves are still being argued out in public.
What it does not cover. Alert and dashboard design, compliance-driven retention for security logs, continuous profiling economics, and vendor feature comparisons; also, deliberately, every source the build environment's network allowlist could not fetch, including the widely reported but here-unverifiable $65M Datadog customer story.
Other field guides
The bill is not a brake
Reconstructs the spend-containment architecture from six published billing incidents (Milkie Way, Troy Hunt, the Netlify and empty-S3-bucket bills, C…
34 sources · 31 organisations · 5 postmortemsWhen owning hardware wins, and when it owns you
Dropbox banked an SEC-audited $74.6M by leaving S3; 37signals cut its bill from $3.2M to $1.3M on the way to deleting its AWS account; and in the sam…
29 sources · 25 organisations · 3 postmortemsZone-local traffic and the cross-zone bill
Spreading across three availability zones is billed twice: in standing idle capacity, and at a cent per gigabyte for every crossing. This guide recon…
34 sources · 21 organisations · 4 postmortems