Behaviour Inventory

A replacement for a 16-year-old claims system passes every acceptance test and goes live. Six weeks later the reinsurance return it produces is rejected by finance, and nothing in the test pack was wrong. The test pack was derived from the documentation, and the documentation describes what the system was commissioned to do in 2009 - not what it does after a decade of fixes, regulatory patches and one-off accommodations.

A behaviour inventory closes that gap by enumerating observable behaviour rather than intended behaviour: every file written, every report produced, every number another system reads, every identifier allocated, every rule applied on a schedule. Each entry has a named consumer, or it is flagged as having none.

The entries with no named consumer are the cheapest finding an assessment can produce, because they are retirement candidates - and retiring an obligation is the only move that reduces what a replacement has to rebuild.

Why it matters

Replacement scope is the single largest driver of programme cost, and it is set by what the system is obliged to keep doing. An inventory makes that list explicit and therefore negotiable. Without one, scope is discovered incrementally during a parallel run, which is the most expensive place to find it.

The second reason is that the missed behaviours are systematically the dangerous ones. Frequent, visible behaviour is in the documentation and in everyone's head. What is missed is infrequent and invisible: the annual regulatory extract, the gapless sequential identifier a ledger depends on, the field another team parses out of a filename, the overnight file whose arrival time is itself a signal. Each of those fails after cutover, in front of an auditor.

Third, it converts assessment from opinion into artefact. Two architects will disagree about whether a system is replaceable; they will not disagree about a list of 240 outputs with 180 named consumers.

Implementation patterns

Build it from four sources that between them name most real consumers, and that do not require interviewing anyone:

• Scheduled jobs: the crontab, the scheduler, the batch definitions. Every periodic obligation is here, including the annual ones that no monthly observation window will reveal.
• Outbound artefacts: files written, reports generated, messages published, with destinations. File names and directories usually identify the consumer.
• Database grants and query logs: who reads which table directly. This is where the undocumented dependencies live, and it is the source teams forget because it bypasses the application entirely.
• Inbound interfaces: API access logs and integration configuration, by caller.

Then enrich each entry with the properties a replacement must preserve: frequency, the format and its tolerances, ordering or gaplessness guarantees, timing the consumer relies on, and whether the behaviour is statutory. Mark each one reproduce, change with consumer agreement, or retire. The three-way classification is what makes the inventory a decision document rather than a list.

Pair the inventory with characterisation tests for the behaviours classified reproduce: tests that record what the system currently does, bugs included, so the replacement can be held to it.

Industry example

The practice comes out of legacy-code work popularised by Feathers' Working Effectively with Legacy Code in 2004, where the governing idea is that an untested legacy system's current behaviour is the specification. In large public programmes the consequence of skipping it is visible in the record: the US Government Accountability Office's reporting on the IRS CADE 2 programme, begun in 2009 to replace a 60-year-old master file, describes scope being reduced to core components while the legacy system continued to carry the obligations - which is what happens when the obligation list is discovered rather than negotiated.

Failure scenarios

• The statutory output nobody owns. An annual return assembled from a table no current employee mentions. It is not in the inventory because the interview-based assessment never reached the person who runs it.
• Identifier semantics lost. The old system allocated gapless sequential invoice numbers; the replacement allocates UUIDs. The general ledger reconciliation depends on gaplessness, and nobody wrote that down.
• Timing as an undocumented contract. A downstream team keys off the 02:00 arrival of a file. The new system produces the same file at 05:30 and a different team's SLA breaks.
• Direct database readers. The application is made read-only and switched off; three weeks later a reporting job that queried the database directly fails.
• Inventory as a one-off. Built in month one, never updated, and wrong by month nine because the legacy system kept changing.

Trade-offs

The inventory costs two to six weeks of a senior engineer and an analyst for a substantial system, and it competes with starting the build - which is what the sponsor wants to see. It is also incomplete by construction: logs do not capture a consumer who reads an emailed spreadsheet.

The counter-argument is the cost asymmetry. A missed statutory behaviour is found after cutover, by a regulator or an auditor, and costs an order of magnitude more than the weeks the inventory would have taken. Buy the inventory for the behaviours with external obligations, and accept sampling for the rest.

When not to use it

Do not build a full inventory for a system being rehosted unchanged. If the application binary and its outputs are identical afterwards, the behaviour is preserved by construction and the inventory buys nothing.

Do not build one for a system with a small, well-known and already tested interface - a service with 20 endpoints, a contract test suite and no batch outputs. The inventory already exists; it is the test suite.

And where the replacement is explicitly permitted to change behaviour - a greenfield product with consumers who have agreed to migrate - the inventory's purpose shifts from reproduction to migration planning, and a lighter consumer list is enough.

Interview question

Q: A replacement for a 16-year-old claims system passed every acceptance test and then produced a reinsurance return that finance rejected. How would you have found that gap during the assessment, and what would you have produced?

What a strong answer covers: that acceptance tests derive from documented intent while the obligation derives from observed behaviour · the four evidence sources - scheduled jobs, outbound artefacts, database grants and query logs, inbound interfaces - and why each is chosen · enrichment with frequency, format tolerance, ordering guarantees, timing and statutory status · the three-way classification into reproduce, change with agreement, retire · characterisation tests for the reproduce set · and the honest limits: emailed spreadsheets and human workarounds will not appear, so sample interviews are still needed for the statutory entries.

Quick check

Quiz: Which two categories of behaviour are missing from almost every legacy assessment, and why? Statutory outputs and downstream-consumer quirks - both are infrequent and invisible, so they are absent from documentation, from the test pack and from everybody's recollection.

Flashcard: Which four sources build a behaviour inventory without interviewing anyone? - Scheduled jobs, outbound artefacts and their destinations, database grants plus query logs, and inbound interface logs by caller; between them they name most real consumers, including the annual obligations a 30-day observation window never shows.