Datadog's engineering writing on Husky, its third-generation event store, says the team dual-wrote all data and shadowed 100% of the query load for months before switching a product over, and that the switch raised median latency by a few hundred milliseconds while cutting p95 and p99 by seconds. What did that proof cost, and when is it the wrong way to do a migration?

The situation they were in

Husky is described as an unbundled column store built around commodity object storage, with metadata in FoundationDB and separate Writers, Compactors and Readers. The replaced systems held customer telemetry: data a customer queries during their own incident, where a wrong answer is worse than no answer and nobody will accept "we are migrating" as an explanation.

Datadog's account also gives the duration honestly: over a year and a half from the first line of code to fully migrating one product.

What the proof cost

Dual-writing everything means paying twice for ingestion and twice for storage for the length of the overlap, which for a petabyte-scale event platform is the single largest line in the migration budget. Shadowing 100% of query load adds the whole query compute of the new system on top, for months, serving results nobody sees.

So the cost of the evidence is roughly 2x storage plus 2x ingest plus 1x query compute, sustained, and it buys three things that nothing cheaper buys:

• Correctness under the real query mix, including the pathological queries that only a few customers write and that no synthetic suite contains.
• A latency distribution measured rather than predicted, which is what let them see the trade they were making: a worse median in exchange for a far better tail. That is a decision a product owner must take, and it can only be taken against real numbers.
• A rollback that stays free. While the old path still serves, reverting is a configuration change rather than a project.

The trade they accepted

A few hundred milliseconds of median latency for seconds to hundreds of seconds off p95, p99 and max. For an observability product that is a clear win, because the queries that hurt are the slow ones during an incident, and a user who waits 400 ms instead of 150 ms does not notice. For a transactional API with a 200 ms budget the same trade would be unacceptable. The lesson is not "prefer tails"; it is that the shape of the distribution is a product requirement and must be stated as one.

Where copying this is a mistake

The shadow-everything approach is priced for a system whose correctness is the product. For an internal reporting warehouse, shadowing 1% of queries and the heaviest 20 by cost gives most of the signal for a few per cent of the bill. For anything where the old and new systems share a storage layer, dual-write does not even test the thing you are worried about.

The decision rule: shadow a fraction when you are validating performance; shadow everything when you are validating correctness on inputs you cannot enumerate. And set the overlap's end date in advance with the arithmetic attached, because a dual-write period with no deadline is how a one-year migration becomes a three-year one with two systems to run for ever.

Common weak answers

• "Dual-write briefly, then cut over." A short overlap tests the common path. The queries that break a new engine are rare by construction, so a short window is the one thing that cannot find them.
• "Compare with a synthetic benchmark instead." A benchmark contains the queries you thought of. The whole value of shadowing is the ones you did not.
• "The median regression means the migration failed." It means the engine made a different trade. Measured against what the product sells, it was the right one.