A telemetry product must answer arbitrary filtered queries over hundreds of billions of events retained for 15 months, with sub-10-second response for the last 24 hours. Each event has 40 to 200 fields and cardinality is unbounded. Which storage design fits?
Show the full answer Hide the answer
The deciding property
Retention length multiplied by unbounded cardinality is what settles this. Fifteen months of event data at this volume is a storage problem measured in petabytes, and any design that keeps it on cluster-attached disk ties the cost of storage to the size of the compute fleet. The query pattern is the second fact: queries filter by time first and touch a handful of the available fields, which is the exact access pattern columnar storage is built for.
Datadog's Husky is the documented implementation of this answer: a schemaless column store built around commodity object storage, with writers that read from Kafka and upload files, compactors that rewrite small files into larger ones, and readers that query, all sharing a metadata store and each scaling independently. Events are bucketed by time so that a query prunes to the relevant buckets before scanning anything.
The separation is the point. Ingestion spikes, compaction backlog and query load are three different workloads with three different failure modes, and coupling them to one fleet means a query storm slows ingestion.
Why the other options fail
- A full-text inverted index over every field is the design that makes ingestion, not query, the expensive step. The index approaches the size of the data, and with unbounded cardinality the index grows with distinct values rather than with useful information. It gives fast arbitrary search on recent data and it prices out 15-month retention.
- A real-time OLAP store holding all 15 months is a correct answer to a different question. Stores of that class are excellent for sub-second aggregation over recent high-volume data, and holding the full retention on cluster nodes means the fleet is sized by storage, not by query load, which is the cost structure the object-storage design exists to escape. A common and defensible hybrid is this store for the last day and object storage behind it.
- A relational warehouse with nightly rollups breaks on the requirement that queries be arbitrary. A rollup is an aggregation decided before the question was asked, so the per-customer, per-build, per-region question that an incident actually raises is unanswerable because the dimensions were discarded at write time.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| Retention drops to 7 days | real-time OLAP store | storage fits on nodes and you gain sub-second latency |
| Queries become fixed dashboards | warehouse with rollups | precomputation is cheaper than scanning |
| Volume drops two orders of magnitude | a single columnar database | the operational cost of three fleets stops being justified |
| Full-text search over message bodies becomes the main use | an inverted index over a bounded window | the index pays for itself when search is the product |
When this is the wrong answer
Below roughly a terabyte a day, or with a retention of days rather than months, none of this is warranted. One managed columnar database holds the data, answers the queries and costs one team's attention instead of three fleets' worth. The unbundled design is a response to a storage bill that has outgrown the compute that reads it, and adopting it earlier buys operational surface with no return.
What a strong answer adds
Object storage introduces latency of tens to hundreds of milliseconds per request, so the design only works because time-bucketing and metadata pruning keep the number of files touched small, and because compaction keeps file counts from exploding. Compaction is the part teams underestimate: without it, a query over an hour touches thousands of tiny files written by parallel writers, and the metadata store becomes the bottleneck rather than the bytes.