Reference Data

Definition

Reference data is the set of permitted values shared across systems: country and currency codes, units, product categories, status enumerations, tax rates, organisational hierarchies. Low volume, slow change, high blast radius.

Why it causes trouble disproportionate to its size

• Everyone has a copy, and they drift. One system's country list includes a territory another's does not, and a record fails to match.
• Changes are rare and consequential. A new tax rate, a currency redenomination, a category restructure. When it happens, every system must be updated consistently and usually is not.
• Nobody owns it. It is too small to warrant a team and too shared to belong to one.
• Historical accuracy matters. A transaction from 2019 must be interpreted with the 2019 tax rate, not today's. Systems that overwrite reference data silently rewrite history.

The design that works

A single owner and a single publication point. One system is authoritative; everything else consumes. This is the decision that resolves most of the problems, and it is organisational rather than technical.

Distributed rather than queried. Reference data should be replicated locally — as a file, a cached dataset, or a small local table — not fetched per request. It changes rarely and is needed constantly; a synchronous lookup makes the reference service a dependency of everything.

Versioned, with effective dates. Each entry carries a validity period, so historical records are interpreted with the values that applied at the time. Overwriting is the mistake that quietly corrupts reporting.

Change managed like a release. Notification, a lead time, and a verification that consumers have picked it up. A reference change that lands unannounced breaks things in ways that are hard to attribute.

Backward compatible by default. Add codes; deprecate rather than remove; never reuse a retired code, for the same reason a protobuf field number is never reused.

Failure scenarios

• Every system with its own copy, drifting.
• Overwritten in place, so historical records are reinterpreted and reports change retroactively.
• A code reused after retirement, silently reclassifying old records.
• Fetched synchronously, making the reference service critical.
• No owner, so changes happen ad hoc.

Interview question

"Tax rates change annually and must apply from a specific date. How do you manage that across a dozen systems?"