Dual Read

Dual-write puts data into the new store. It says nothing about whether that data is right, and the defects that matter in a migration are the quiet ones: a timezone applied twice, a rounding rule inherited from a column type, a tombstone that was never copied. A migration that dual-writes and then cuts over discovers those on the day it matters.

Dual read closes that gap. Every production read is answered from the legacy system, and the same read is issued against the new one and compared. The user sees the old answer; the engineering team sees a diff rate on live traffic, with the real distribution of keys, including the pathological ones no test suite contains.

Why it matters

It is the only verification method whose input distribution is production's. A backfill checksum proves the bytes copied. A test suite proves the cases someone thought of. Dual read exercises the new store with the actual mix of keys, access patterns and oddities, and it produces a number - diff rate per endpoint, per partition, per field - that a cutover decision can be made against.

It also converts the cutover from an event into a threshold. Flip reads when the diff rate for that slice has been zero for a defined period, rather than when the plan said so.

Implementation patterns

• Issue the comparison read asynchronously, off the response path, so a slow or failing new store cannot affect users. This is the single most important detail: a synchronous dual read makes the migration's reliability the product's reliability.
• Compare on a canonical form. Normalise field order, numeric precision, null-versus-empty and timestamp representation before comparing, or the diff rate is dominated by formatting and nobody reads it.
• Sample by cost. Compare every read at low volume; at high volume, sample a few per cent with the sample stratified so rare key types are over-represented.
• Record the diff, not just the count. A diff rate with no examples cannot be triaged, and triage is where the actual defects are found.
• Classify expected differences explicitly. When the new system deliberately fixes a legacy bug, that difference must be modelled as expected or it buries the real ones.
• Keep it running after cutover, reversed: read from the new store, compare against the legacy one, for as long as the legacy system is still fed.

Industry example

The most instructive published case is Datadog's account of building Husky, its event store: before switching a product over, the team dual-wrote all data and shadowed 100% of the query load for months, which is dual read at full scale on a system whose correctness is the product. The cost was explicit - double ingest, double storage, and the new system's entire query compute serving results nobody saw - and what it bought was both correctness on the real query mix and a measured latency distribution, which is how the team could see that it was trading a few hundred milliseconds of median for seconds off p95 and p99.

Failure scenarios

• Synchronous comparison on the request path, so the new store's p99 becomes the product's p99 and the migration causes the incident it was meant to prevent.
• A diff rate dominated by formatting noise, after which the metric is ignored and a real defect hides in it.
• Comparing aggregates rather than records, where two offsetting errors cancel and every total matches.
• Comparison traffic doubling load on the legacy system, which is often the thing least able to take it.
• No triage owner. A diff queue with nobody rostered becomes a dashboard of unread evidence.
• Sampling that excludes the interesting keys, so the one tenant with 40 million rows is never compared.

Trade-offs

When not to use it

Do not dual-read when the new system's read path is identical to the old one - a like-for-like database upgrade or a lift-and-shift - because a checksum over the data proves everything the comparison would. Skip it for datasets small enough to compare exhaustively offline, where a one-off full reconciliation is both cheaper and stronger. And do not build it for a migration whose output is not deterministic, such as a ranking or recommendation change: there the comparison is an evaluation problem with its own methodology, not a diff.

Interview question

Q: You are three months into migrating an orders service to a new datastore. Dual-write has been running cleanly and the team wants to cut reads over next sprint. What evidence do you require first, and how do you gather it without risking production?

What a strong answer covers: requiring a measured diff rate on production reads rather than a green test suite; issuing comparison reads asynchronously off the response path; canonicalising before comparison and modelling expected differences; sampling stratified towards rare keys; insisting on per-partition or per-tenant diff rates so cutover can be incremental; naming a triage owner; and keeping the comparison running in reverse after cutover while the legacy system is still fed.

Quick check

Quiz: Why must a dual read be issued asynchronously? Otherwise the new store's latency and failures become the product's, and the verification mechanism becomes an availability risk.

Flashcard: What does dual read prove that a backfill checksum does not? That the new system's read path returns the right answers for production's actual key distribution, not merely that the bytes were copied.