Rollup Policy

A real-time analytical store's cost is dominated by retention and cardinality, and both are far cheaper to decide at ingestion than to change later. The asymmetry is the point: raw data that was never retained cannot be rolled up differently afterwards, so the policy forecloses options in one direction only.

A rollup policy states, per dataset: what granularity is kept, for how long, at what dimension cardinality, and what the data collapses into as it ages.

Why it must precede ingestion

• Rollups defined at ingestion are cheap; backfilling them means reprocessing history you may no longer have.
• Retention is the dominant cost line, and extending it later is easy while recovering discarded data is impossible.
• Cardinality is the most common cause of unexpected cost and latency, and it must be tested against the real dimension set rather than a sample — a dimension that looks bounded in staging is frequently unbounded in production, and a single user-identifier or URL dimension can multiply series count by orders of magnitude.

Implementation patterns

• Tiered granularity by age: raw for a short recent window, minute rollups for weeks, hourly or daily beyond that. Query patterns follow the same shape, so this rarely costs anything anyone notices.
• Rollups computed at ingestion, not on read, which is where the performance of these stores usually comes from.
• A dimension allowlist, with high-cardinality attributes excluded from indexed dimensions and kept in a separate raw store if needed at all.
• Time-based partitioning with automatic expiry, so retention enforces itself rather than depending on a cleanup job someone disables.
• Query resource isolation, so an expensive analytical query cannot degrade ingestion — the single most important operability property of these stores, and the one that separates them in practice.
• A stated boundary against the warehouse: recent-window speed and immediate queryability here, complex historical analysis and large joins there, with one direction of derivation.

Industry example

Observability and edge-analytics platforms ingesting very high event volumes converge on the same shape: approximate, heavily pre-aggregated recent data queried in milliseconds, with raw retention measured in days rather than months, and long-horizon analysis pushed to cheaper columnar storage.

The mistake that recurs is treating the real-time store as a general warehouse because it happens to speak SQL. It is optimised for high-throughput ingestion with immediate queryability, which is a different problem from flexible historical analysis, and using it for both produces a system that is expensive at one and bad at the other.

Failure scenarios

• Retention chosen "for now" and never revisited, becoming the largest line on the bill.
• Unbounded dimensions, where cardinality growth degrades cost and latency simultaneously.
• Rollups deferred, then found to be un-backfillable because the raw data has expired.
• No query isolation, so an analyst's query causes an ingestion lag incident.
• The store used as a warehouse, producing slow complex queries and a recurring argument about the platform choice rather than about the policy.

Trade-offs

Aggressive rollup discards detail permanently, and the question it forecloses is always one nobody anticipated. Conservative retention preserves optionality and costs continuously.

The workable middle is short raw retention feeding a cheap durable log of the original events elsewhere: the real-time store stays lean, and the ability to recompute a different rollup survives in object storage at a fraction of the cost. Retain raw cheaply; index selectively.

Interview question

"You are designing a real-time metrics store for a platform emitting a million events a second. What do you decide before you ingest the first event, and why can those decisions not wait?"