Unbundled Event Store

A conventional analytical store couples storage to compute: data lives on disks attached to the nodes that query it, so retaining more months means running more nodes, whether or not anyone queries the old data. At telemetry volumes, where retention is measured in quarters and ingestion in millions of events a second, that coupling is what makes the bill unbearable.

The unbundled design breaks it. Data lives in a columnar format on commodity object storage. Writers, compactors and readers are separate services that share only the storage and a metadata store describing which files exist and what they cover. Each scales for its own load.

Why it matters

The three workloads have nothing in common. Ingestion is steady and spiky at the edges; compaction is batch-shaped and deferrable; querying is bursty and latency-sensitive. Running them on one fleet means sizing for the sum of their peaks and accepting that a query storm slows ingestion, which in a telemetry product means the customer loses the data about the incident they are investigating.

Separation also changes what failure looks like. A compactor outage degrades query performance gradually over hours rather than stopping anything, which is a far better failure than the coupled equivalent.

Implementation patterns

• Bucket by time first. Queries filter on time, so placing events into time-dimension buckets lets the metadata store prune the file set before a byte is read.
• Columnar files with row-group structure, so a query touching 5 of 200 fields reads only those columns. Datadog's Husky uses a format described as similar to Parquet but tuned for observability data.
• A transactional metadata store holding the file catalogue, so writers can commit new files atomically and readers always see a consistent set. Husky uses FoundationDB for this; the requirement is transactions and low-latency lookups, not analytics.
• Compaction as a first-class service rewriting many small files into fewer large ones. Without it, a query over an hour opens thousands of objects, each costing a request round trip of tens to hundreds of milliseconds.
• Schemaless ingestion with column inference, because telemetry fields are not declared in advance and unbounded cardinality is the norm rather than an abuse.

Industry example

Datadog's Husky, described across a series of engineering posts from 2022 onwards, is the documented implementation: writers consume from Kafka, buffer, upload files to blob storage and commit their existence to the metadata store; compactors rewrite them; readers query. The published figures describe query access to event volumes at the hundred-trillion scale, with the three roles scaled independently on shared S3 storage.

Failure scenarios

• Compaction falls behind. Query latency rises with no change in data volume or query pattern, and the metadata store becomes the bottleneck rather than the bytes. The signal is files-per-bucket and compaction backlog age, not CPU.
• Metadata store saturation. It is the one shared, stateful, transactional component; when it is slow, every role is slow, which makes it the piece to over-provision.
• Object storage request limits. Cost and latency are driven by request counts as much as bytes, so a query pattern that touches many small objects can be expensive while reading little data.
• Write amplification from compaction, which rewrites the same bytes several times as files are merged upward, and can exceed the original ingest volume if the tiering policy is naive.

Trade-offs

Gains: storage cost decoupled from compute, retention extended by changing a lifecycle policy rather than a fleet size, and three workloads that can no longer starve each other. Pays: object storage latency of tens to hundreds of milliseconds per request, which rules out sub-100ms interactive queries without a hot tier in front; a metadata store to operate as a critical dependency; compaction as a permanent background cost; and three services to run instead of one.

When not to use it

Below roughly a terabyte a day, or with retention measured in days rather than months, one managed columnar database is the right answer and this design buys operational surface with no return. It is also wrong when the product requires sub-second interactive queries over the whole retention: object storage cannot deliver that, and the common resolution is a hybrid where a real-time analytics store holds the most recent day and the unbundled store holds the history. The rule: unbundle when the storage bill has outgrown the compute that reads it. Until then, coupling is a feature, because it is one system to operate.

Interview question

Q: Your object-storage-backed event store's p95 query latency has doubled over six weeks. Ingest volume is flat, query volume is flat, and no query has changed. What do you investigate, and what is the most likely cause?

What a strong answer covers: files touched per query as the primary metric, not bytes scanned · compaction backlog age and files-per-time-bucket · the mechanism whereby more small files means more object requests and more metadata lookups for the same data · checking whether writer count or flush frequency changed, since more parallel writers produce more small files · the fix being compaction throughput rather than reader capacity · and the monitoring gap that let it develop over six weeks.

Quick check

Quiz: Why is compaction a service rather than housekeeping in this design? Because query cost is driven by the number of files opened, so falling behind on compaction degrades query latency independently of data volume.

Flashcard: What does unbundling actually decouple? — Retention cost from query-fleet size: keeping another six months changes a storage lifecycle policy instead of adding nodes.