A team wants to debug failures nobody predicted, using wide high-cardinality events rather than pre-aggregated metrics. How does the storage design differ, and how must sampling preserve rare errors?

Why pre-aggregated metrics cannot answer unknown questions

A metric is an aggregation decided at write time. When you emit http_requests_total{status, endpoint}, you have committed to the questions answerable later: you can ask about status and endpoint, and nothing else.

The dimensions you did not think to add are permanently unavailable. When the incident turns out to affect one customer, on one device type, in one region, on one build, the metric cannot answer it — the information was discarded at write time, and no amount of querying recovers it.

Adding dimensions does not fix this, because metric cost is multiplicative in cardinality. A time series exists for every combination of label values, so adding customer_id with 50,000 values multiplies the series count by 50,000. This is why traditional metrics systems forbid high-cardinality labels, and why they are structurally unable to answer per-customer questions.

What a wide event is

One record per unit of work — typically per request — carrying every field that might matter: identifiers, customer, region, build, feature flags, cache hit or miss, downstream latencies, queue depth on arrival, user agent, error detail. Fifty to several hundred fields is normal.

Nothing is aggregated at write time. Aggregation happens at query time, from the raw events, so any question expressible over the fields is answerable — including ones nobody anticipated. That is the entire proposition: debugging unknown-unknowns requires that the data was not summarised before the question existed.

How the storage differs

• Columnar, not time-series. Queries touch a few of many fields, so column-oriented storage reads only the needed columns — the same reasoning as an analytical database, applied to telemetry.
• No index on high-cardinality fields. Indexing every field is prohibitive; instead the system is built for fast full scans over compressed columns within a time range, with the time range as the primary pruning mechanism.
• Cardinality is free. A customer_id column with a million distinct values costs the same as one with three — it is a column, not a series. This is the fundamental difference, and it is what makes the model viable.
• Retention is shorter, because raw events are large. Weeks, not years, with aggregates derived for long-term trends.

Honeycomb's Retriever is the well-documented implementation of this design.

Sampling that preserves rare errors

Storing every event is uneconomic at volume, and uniform random sampling is the wrong answer: it preserves the common cases you already understand and discards the rare ones you need.

• Head-based sampling by key: derive a rate from the event's characteristics. Successful, fast, routine requests at 1 in 1,000; errors at 1 in 1; slow requests at 1 in 1. Sample the boring traffic aggressively and keep the interesting traffic entirely.
• Dynamic sampling that adjusts rates from observed frequency, so a newly-rare condition is automatically kept at a high rate without anyone configuring it. This is what handles the cases nobody anticipated, which is the whole point.
• Tail-based sampling decides after the trace completes, so the decision can consider the outcome — the correct semantics, at the cost of buffering every trace until it finishes, which is a real infrastructure burden.
• Record the sample rate on every event, so queries can reconstruct true counts by weighting. Without this, sampled data silently understates volume and every count is wrong.
• Never sample per-log-line; sample per-trace or per-request, or you get partial traces that are worse than none.

The honest trade-off

Wide events cost more per unit of data and answer far more questions. Metrics remain the right tool for long-retention trends, for alerting on known conditions, and for dashboards — they are cheap, fast, and sufficient for things you already know to watch.

The mature position is both: metrics for alerting and trends, wide events for investigation. The failure is having only metrics and being unable to explain an incident, which teams discover during the incident rather than before it.