Datadog described Husky in 2022 as its third-generation event store: writers consume from a queue, write schemaless columnar files to object storage, and commit the presence of those files to a transactional metadata store, with ingestion, storage and query scaled independently. What forced that shape rather than a conventional search cluster on local disks, and where would copying it be a mistake?

The situation they were in

Telemetry has four properties that fight a single clustered index. It is schemaless, because customers attach arbitrary tags and fields. It is write-dominated and cannot pause, because the ingest firehose has no back button. It is multi-tenant, so one customer's expensive query sits next to another's ingestion. And the query mix is split between analytical aggregation over billions of events and needle-in-haystack search for one event.

A search cluster with data on node-local disks couples all four. Retention becomes a capacity project, because keeping 90 days means buying and rebalancing 90 days of replicated SSD. Adding query capacity means adding storage you did not need. A heavy query competes with the write path for the same page cache and the same CPU, and an ingestion gap is not recoverable while a slow query is.

What they chose, and why it fits

Separate the three paths. Writers read from the queue, produce columnar files, and put them in object storage; the fact that a file exists is committed to a transactional store, which is what lets exactly-once ingestion be expressed as a transaction rather than reconstructed by deduplication later. Query nodes are stateless readers.

The fit is economic before it is architectural. Object storage costs roughly an order of magnitude less per byte than replicated local SSD, so retention turns from a capacity decision into a pricing decision, which is exactly what a vendor selling retention tiers needs. Query capacity scales with query load and cannot starve the writers.

What it cost them

Every read is a network read, so a small point lookup is slower than it would be against a local index. Freshness and scan efficiency pull in opposite directions: ingest wants small files written often, queries want large files, so compaction becomes a permanent, continuously running subsystem with its own capacity, its own failure modes and its own cost line. Datadog wrote a separate post on compaction alone, which is a fair signal of the weight involved. The metadata store also becomes the correctness bottleneck for the whole system: if the record of which files exist is wrong, the data is intact and unreadable.

When not to copy it

Below a few terabytes of retained logs, a single managed search cluster is simpler, faster to query and cheaper in engineer-months. The unbundled design pays only when two conditions both hold: storage volume, rather than query concurrency, is the dominant cost, and there is a team to own compaction as a product. A four-engineer platform team adopting this shape buys a distributed transaction problem and a small-file problem in exchange for a storage saving that a retention policy would have delivered for free.

The transferable move is not the architecture. It is the question Husky answers: which of ingest, storage and query is your binding constraint, and are you currently forced to buy all three together?