Datadog has published the design of Husky, its third-generation event store, which splits writers, readers and compactors into independently scaled services over a shared metadata store and commodity object storage rather than running one storage tier. What problem forces that separation, and where would copying it be a mistake?

The situation they were in

Observability data has three properties that fight each other: it arrives continuously and must never be refused, it is queried in bursts with a human waiting, and it is schemaless and high-cardinality — customers invent tag keys, so the column set is not known in advance and per-tenant cardinality has no ceiling.

Those three workloads have opposite operational shapes. Ingestion is throughput-bound and must be over-provisioned. Query is latency-bound and bursty. Compaction is a background rewrite that is throughput-bound and interruptible. In a single storage tier they share memory, CPU and a failure domain, so a query storm starves ingestion and a compaction backlog slows queries.

What they chose

Per Datadog's engineering posts on Husky: writers read from Kafka, buffer, write columnar fragments to blob storage and then commit the existence of those files to a metadata store built on FoundationDB. Readers resolve queries through the same metadata. Compactors merge small fragments into larger ones using streaming k-way merges with bounded row groups so memory stays flat, applying size-tiered and locality-based compaction within time buckets, and issuing about one read request per input fragment so the cost scales with fragment count rather than with bytes.

The fragment format is columnar and Parquet-like — one row group, many pages — but built for this data rather than adopted wholesale.

Why it fit their constraints

The metadata store is what makes the split possible. Object storage offers no multi-object transaction, so "the file exists" and "the file is part of the table" have to be separated. Putting the second fact in a transactional store gives atomic commits, lets readers see a consistent view without coordinating with writers, and lets each role scale on its own signal: ingestion on incoming volume, query on concurrency, compaction on backlog.

What it cost them

A bespoke columnar format that no other tool reads. A FoundationDB cluster to run, which is a specialist operational skill. And compaction promoted from a chore to a production service with its own backlog metric and its own on-call — because when compaction falls behind, query cost rises continuously and there is no error to page on.

When copying it would be wrong

Almost everywhere. The design is justified by one specific property: schemaless data with unbounded per-tenant cardinality, at a volume where fragment counts per query run into the thousands. If your schema is known, an open table format with a managed engine gives the same writer/reader/compactor separation, the same atomic commit through a catalogue, and the same object-storage economics — without a format only you can read.

The decision flips when the general-purpose table format's own metadata becomes the bottleneck, which shows up as planning time dominating query time on tables you have already compacted. That is a measurement, and it is the one to take before anyone proposes building a storage engine.