Datadog published Husky in 2022, a third-generation event store built on commodity object storage with FoundationDB holding the metadata, replacing an architecture backed by local SSDs. What forced the change, what did it buy economically, and what did it cost?

The situation they were in

A telemetry store serving logs and traces has two properties that break SSD-backed designs. Retention is long and mostly cold, so stored bytes grow without bound while the fraction queried in any hour is tiny. And the tenant population is enormous and heterogeneous, with schemas and query patterns nobody can provision for individually. Coupling storage capacity to query capacity means buying query hardware to hold data nobody is querying.

What they chose

Husky, described in Datadog's engineering post of May 2022, separates the system into three roles: writers that consume from Kafka and upload a custom columnar format to blob storage, readers that scan files in remote storage and return partial aggregates, and compactors that merge small files into large ones, which is LSM-tree compaction relocated onto object storage. Metadata sits in FoundationDB, chosen for strictly serialisable transactions; the bytes sit in S3. The store is schemaless, with fields indexed at query time.

Why it fits their constraints

• Retention is priced at object-storage rates, which are roughly an order of magnitude below provisioned SSD per byte, and it is decoupled from the machines that answer queries.
• Read and write capacity scale independently of stored volume, so a tenant with a year of retention and one query a week costs nearly nothing to keep.
• Schemaless with query-time indexing means you never pay to index a field no one queries, which is the dominant waste in a fixed-schema telemetry store facing tens of thousands of tenant shapes.
• Stateless writers and readers mean capacity is elastic and a node loss is not a data event.

What it cost them

Median latency got worse. Datadog reported that tail latencies (p95, p99, max) improved substantially after the migration while the median rose by a few hundred milliseconds, because a read now crosses the network to object storage instead of touching a local SSD. They judged that trade acceptable: a store like this is bought for its worst case, not its typical case.

The second cost is compaction. Streaming ingestion to object storage produces many small files, and small files destroy scan performance and multiply request charges. Compaction is therefore not an optimisation but a load-bearing service, and it is itself a distributed system with its own scaling, cost and failure modes.

Where copying it would be a mistake

At a volume one team generates, a single indexed store on local disks is cheaper, faster at the median, and has no compaction service to run. The object-storage design earns its keep when stored bytes vastly exceed queried bytes, when retention is measured in months, and when tenant shapes are too varied to schema. Adopt the idea — separate the bytes from the query fleet — long before you adopt the architecture.

When this is the wrong answer

If your telemetry fits in a store you can afford to keep hot, the median-latency tax buys nothing. The signal to move is storage cost growing faster than query cost, sustained over a couple of quarters.