An error-tracking platform receives millions of events, many of them the same underlying problem with different stack details. How should grouping, deduplication, high-cardinality metadata and retention be designed?

The core problem

Raw error events are nearly useless and aggregated errors are the product. Ten thousand occurrences of one bug is one thing to fix, and a platform that presents ten thousand events has transferred the aggregation work to the user at the worst possible moment.

But grouping is a heuristic, and both failure directions are damaging: over-grouping merges distinct bugs and hides one behind another; under-grouping creates thousands of issues for one bug and drowns the signal.

The grouping design

• A fingerprint from stable attributes — exception type, the top frames of the stack that belong to application code rather than to libraries, the code location. Variable data (a specific ID, a timestamp, a memory address) must be excluded or every occurrence is unique.
• Normalise before fingerprinting: strip numbers from messages, collapse anonymous function names, remove file paths that vary by deployment.
• Let users override the fingerprint, because the heuristic will be wrong for their code and a platform that cannot be corrected is a platform that is wrong permanently.
• Regroup carefully. Changing the grouping algorithm splits or merges historical issues, which breaks everything a user has triaged. Versioned grouping applied only to new events is the safe path.

Handling the volume

• Sample at the client for high-frequency errors, sending a proportion with a recorded multiplier rather than every occurrence. The thousandth identical event adds nothing.
• Rate limit per project, so one broken deployment cannot consume a customer's entire quota in minutes — and communicate it clearly, because silent dropping during an incident is exactly when users need the data.
• Keep full detail for a sample and aggregate counts for the rest. The full context of ten occurrences plus an accurate count of a million is more useful than partial detail on everything.

The retention decision

Errors follow a sharp relevance curve: an error from today is being worked on, one from last month is either fixed or accepted. Short retention for full event detail and long retention for aggregate counts and trends matches how the data is used, and it is far cheaper than uniform retention.

The metadata trade

High-cardinality context — user, release, device, feature flag state — is what makes an error actionable, and it is also what makes storage expensive. The resolution is that it lives on sampled events rather than in an index over all events, so it is available when you open an issue and not paid for on every occurrence.