A replacement for a 16-year-old claims system passed every acceptance test and went live. Six weeks later the reinsurance return it produced was rejected by finance, and nothing in the test pack was wrong. An interviewer asks how you would have found that gap during the assessment. What do you say?

What the interviewer is testing

Whether you understand that a legacy system's specification is partly executed by people. Sixteen years of defects that were never worth fixing have been absorbed by someone downstream: a reconciliation spreadsheet, a monthly correction journal, a rule one analyst applies by hand to any claim over a threshold. Those corrections are requirements. They are not in the code, not in the test pack, and not in the documentation, so a replacement validated against the code is validated against an incomplete system.

The weak candidate treats this as a testing failure. It is an assessment failure, and it is the single most reliable way for a technically correct replacement to produce a wrong business answer.

The clarifying questions that change the answer

• Which recurring outputs does this system produce, and who receives each one? Not interfaces: outputs. Reports, files, journals, extracts, statutory returns. A sixteen-year-old core system typically has twenty to sixty, and a handful are annual.
• Between the system and the recipient, does anybody change the numbers? Ask the recipient, not the system's owner. The system's owner believes the output is correct, which is why the correction exists downstream.
• What does the recipient do when the output is wrong? If the answer is a documented procedure, you have found a compensating control. If the answer is "it is never wrong", ask to watch one period close.
• What was the last thing this system got wrong that nobody fixed? The people who answer this are in operations and finance, not engineering — the defect was closed as working-as-designed in 2014 and the correction has run every month since.

How a strong answer finds it

Walk the outputs backwards from their consumers, and require a named human for every recurring output who has confirmed, in writing, that they change nothing before passing it on. Every "actually, I do adjust the…" is a requirement with an owner, a frequency and an acceptance test. A two-week assessment of a core system that produces thirty outputs can complete this; it takes a dozen conversations, not a code audit.

Then use a parallel run to catch what the interviews missed, and compare the output the business consumes, after the human step, not the output the system emits. A parallel run on raw system output will agree perfectly and prove nothing, because both systems are wrong in the same place the human was fixing.

Common weak answers

• "We would write more tests." Tests encode the behaviour you know about. The gap is behaviour nobody wrote down, and no amount of coverage against the old code finds a correction applied outside it.
• "We would do a longer parallel run." Useful, and it fails if the comparison is at the wrong boundary or if the differing output is quarterly and the run is six weeks.
• "The business should have told us." They did not know it was unusual. A control that has run every month for nine years is not experienced as a workaround.

What a strong answer adds

Compensating controls are also the modernisation benefit nobody prices. Each one is a recurring labour cost and an audit finding waiting to happen, and quantifying them — 8 hours a month of senior finance time, one person who cannot take leave at quarter end — is frequently the strongest number in the business case. The assessment that finds them protects the cutover and funds the programme at the same time.

The decision rule is worth stating plainly: walk every output where a person sits between the system and its use, and skip the walk where the chain is machine-to-machine in production. There, no compensating control can exist and the risk lives in data quality instead, so the assessment should be about contracts and volumes rather than conversations.