Wide Event

Traditional telemetry decides at write time what will be askable later. A metric commits to its label set; a log line commits to its message. Anything not captured in that decision is discarded, and the discarding is irreversible.

A wide event inverts the order. One record per request, job or transaction, carrying everything known about it — identifiers, customer, region, build version, active feature flags, cache outcomes, per-dependency latencies, queue depth on arrival, result size, error detail. Fifty to several hundred fields is normal, and no aggregation happens until someone asks a question.

Why it matters

Incidents are defined by the questions you did not anticipate. The failure affects one customer, on one device type, on one build, behind one feature flag, in one region — a conjunction nobody would have thought to pre-aggregate, and one that a wide-event store answers in seconds because every field is present on every record.

The alternative is the familiar incident pattern: the dashboards show that something is wrong and cannot show what, so the investigation proceeds by deploying additional instrumentation and waiting for the problem to recur. That loop costs hours and only works if the problem is reproducible.

Implementation patterns

• One event per unit of work, emitted at completion, so the record can include the outcome and total duration — not one event per log statement, which fragments the context that makes the model work.
• Accumulate context throughout the request in a per-request bag, and emit once at the end.
• Include high-cardinality identifiers freely — customer, user, request, trace, build, host. This is the point of the model, and it is precisely what a metrics system cannot do.
• Include the inputs to decisions, not only the outcomes: which flag was on, which branch was taken, which cache was consulted and whether it hit.
• Record the sample rate on the event, so query-time weighting reconstructs true counts.
• Columnar storage with time-range pruning, since queries touch a few of many fields and scanning compressed columns beats indexing all of them.
• Keep metrics alongside, for alerting and long-retention trends. The two are complementary, not competing.
• Link to traces via a shared identifier, so an aggregate finding drills into individual examples.

Industry example

Honeycomb's product is built on this model, with its Retriever columnar store designed for fast scans over arbitrarily wide, high-cardinality events rather than for indexed lookups. The published rationale is explicit: high-cardinality dimensions are what make debugging possible, and time-series storage makes them prohibitively expensive, so a different storage engine is a prerequisite rather than an optimisation.

The pattern is not vendor-specific. The "canonical log line" — one wide structured record per request — is a long-standing practice at organisations including Stripe, and is achievable on ordinary infrastructure with a columnar analytical store as the backend.

Failure scenarios

• Emitting many narrow events instead of one wide one, losing the correlation that makes the model work.
• Wide events stored in a time-series system, producing a cardinality explosion and a very large bill — the storage engine is not an implementation detail here.
• Uniform sampling, discarding the rare events the model exists to preserve.
• Sample rate not recorded, so all counts are silently wrong.
• Fields added inconsistently across services, so a query that works for one service returns nothing for another.
• Sensitive data included by default, since a record designed to hold everything will hold personal data unless someone decides otherwise.
• Retention set as though these were metrics, producing unaffordable storage — raw events are large and belong on weeks, not years.

Trade-offs

Wide events cost more per unit of data than metrics and have shorter practical retention, so they are not a replacement for long-term trend storage or for cheap alerting. A team that replaces its metrics with wide events discovers that alerting is now expensive and year-over-year comparison is impossible.

They also require discipline about what goes into the event, both for cost and for privacy: a field added casually is a field stored on every event forever, and a personal identifier added casually is a compliance problem distributed across the telemetry estate.

The trade is storage cost and retention in exchange for the ability to answer questions that were not anticipated. For a system whose failures are well-understood and repetitive, metrics suffice. For a system complex enough that its interesting failures are novel, wide events are the difference between diagnosing an incident and waiting for it to recur.

Interview question

"Our dashboards say error rate is up 0.3% and we cannot work out who is affected. Tell me why the metrics cannot answer that, what you would have had to instrument beforehand, and what it would cost us to keep that data for every request."