A platform needs sub-second analytical queries over recent high-volume event data. What store characteristics matter?
Show the full answer Hide the answer
The characteristics that matter
1. High ingestion throughput with immediate queryability. The defining requirement: data must be searchable within seconds of arrival, which excludes stores that batch into large immutable files before indexing.
2. Columnar storage with aggressive compression, since analytical queries read few columns over many rows.
3. Pre-aggregation or rollup support, so common queries read summarised data rather than raw events. For high-volume telemetry this is usually where the performance comes from.
4. Time-based partitioning with automatic expiry, because the data is overwhelmingly time-series and old data is either rolled up or dropped.
5. Cardinality tolerance. These stores are frequently used with high-dimensional data, and they differ enormously in how gracefully they handle it. Cardinality is the most common cause of unexpected cost and latency, and it must be tested against the real dimension set.
6. Tiered storage, so recent data is on fast media and older data on cheaper storage, queried more slowly.
The trade against a general warehouse
A real-time analytical store buys ingestion latency and query speed over recent data, and gives up general SQL flexibility, joins across large datasets, and long-history analysis. Complex historical analysis belongs in the warehouse; the real-time store serves the recent window.
Running both is normal; the discipline is a clear boundary and one direction of derivation, so they do not diverge and consumers do not have to choose.
The properties that decide operability
- How rollups are defined and recomputed when a definition changes.
- What happens to late-arriving data after a period has been rolled up.
- Rebalancing behaviour when capacity changes.
- Query resource isolation, so one expensive query does not affect ingestion — which is the failure mode that turns an analytics problem into a data-loss problem.
The design guidance
Decide the retention and rollup policy before ingesting anything. Retention is the dominant cost driver, and rollups are far cheaper to define at ingestion than to backfill afterwards — while raw data that was never retained cannot be rolled up differently later.