Comparison Harness

A general ledger replacement runs in parallel for five months. The nightly check compares daily totals per account class and they match to the penny every night. The replacement is wrong on thousands of individual records, because two defects of opposite sign in the same class cancel, and a total is the easiest aggregate to make match.

A comparison harness is the difference between a parallel run that proves something and one that provides comfort. It is a system in its own right: it decides what is compared, at what granularity, against what tolerance, and what happens to each difference it finds.

Why it matters

A parallel run's entire value is in its diffs, and diffs need a pipeline rather than a report. Without one, the output is a nightly email that is green until it is red, and when it goes red there is no classification, no owner and no history. With one, the output is a diff rate per field per day, a queue of examples, and a trend that tells a steering committee whether the replacement is converging.

It is also the only practical way to capture undocumented behaviour. The legacy system's edge cases are documented in its own source and nowhere else, and a harness running real inputs through both is how they are discovered before cutover rather than after.

Implementation patterns

• Compare at record level, not only in aggregate. Every record at low volume; a stratified sample at high volume, deliberately over-weighting the rare - back-dated entries, corrections, foreign currency, the largest accounts.
• Canonicalise before comparing. Field order, numeric precision, null versus empty, timestamp representation and string trimming, or the diff rate is dominated by formatting and nobody reads it.
• Write tolerance rules per field, with a justification. "Interest to ±0.01" is a materiality decision that belongs to the business, not a technical default. A tolerance with no stated reason is a place for defects to hide.
• Model expected differences explicitly. Where the new system deliberately corrects a legacy bug, that difference is expected and must be classified as such, or it buries the unexpected ones.
• Triage daily, with an owner. Each diff gets a class (known legacy bug, mapping defect, tolerance, data quality, genuine regression), and the class is what the trend is reported on.
• Publish a diff rate as the project's headline metric, per module, and make cutover conditional on it rather than on a date.
• Keep it running after cutover, comparing the new system against independently recalculated expectations, because a plausible-but-wrong output is found by comparison or not at all.

Industry example

The strongest published example of this discipline at scale is Datadog's Husky migration, where the team dual-wrote all data and shadowed 100% of the query load for months before switching a product over - comparison on the real input distribution rather than on a test suite, with the trade it revealed (worse median, far better tail) made explicit. In regulated replacements the harness is often mandated: ledger and policy-administration programmes are routinely required to demonstrate a period of record-level agreement, which is why the tolerance rules and their justifications end up as audit artefacts rather than engineering notes.

Failure scenarios

• Aggregate-only comparison, where offsetting errors net to zero and every published total agrees.
• A diff queue nobody owns, so a defect found on Monday is understood a fortnight later and the backlog becomes the reason to stop looking.
• Tolerances widened to make the report green, which is the single most common way a parallel run stops being evidence.
• Comparison load on the legacy system that pushes it over its own capacity, so the harness causes incidents.
• Sampling that misses the interesting population, usually the largest tenant or the oldest records.
• Unmodelled deliberate differences, so the diff rate never reaches zero and the team learns to ignore a permanently non-zero number.

Trade-offs

When not to use it

For a like-for-like migration where the logic is unchanged - a database version upgrade, a lift-and-shift, a re-host - a checksum over the data and a smoke test prove as much, and a harness is expensive theatre. It is also the wrong tool when outputs are legitimately non-deterministic, such as a ranking or recommendation change: there the question is which output is better, which is an evaluation problem with its own methodology, not a diff. And for a small dataset, an exhaustive one-off reconciliation is both cheaper and stronger than a continuous harness.

Interview question

Q: A five-month parallel run of a billing replacement has matched nightly revenue totals every night, and the programme wants to cut over. What do you ask for before agreeing?

What a strong answer covers: recognising that matching totals cannot rule out offsetting errors; asking for record-level diff rates per field with examples, and for the sampling design; reviewing the tolerance rules and who justified each one; checking that deliberate differences were modelled rather than tolerated; asking for the triage history and whether the diff rate is converging; requiring the rare populations (corrections, back-dated entries, largest accounts, foreign currency) to be over-sampled; and making cutover conditional on the diff rate per module rather than on the elapsed duration of the run.

Quick check

Quiz: Why can a parallel run match every nightly total and still be wrong? Offsetting errors of equal size in opposite directions cancel in any aggregate, so only record-level comparison detects them.

Flashcard: What makes a tolerance rule dangerous? Having no stated justification - an unexplained tolerance is where defects hide, and widening tolerances to make the report green is how a parallel run stops being evidence.