Data Contract

Analytical pipelines routinely read directly from a producing service's database or from a raw ingested copy of it. That table is the service's internal implementation, and its owner has made no commitment about it — so a column rename during ordinary refactoring silently breaks a dozen downstream pipelines and several dashboards.

The producer discovers this when someone complains, days later. They had no way of knowing, because nothing recorded that anyone was reading it.

A data contract makes the dependency explicit: a defined schema, defined semantics, defined quality guarantees, a defined change policy, and a known owner on each side.

Why it matters

The failure is asymmetric and misattributed. The producer's change was correct and reasonable; the breakage occurs in someone else's system; and the person who can prevent it has no visibility of the consumers. No amount of care by either party fixes a dependency neither can see.

It also enables the producer to change safely, which is the benefit that gets producers to agree. A contract tells them precisely what is committed and what is free to change — and a field no contract mentions can be altered without consultation, which is otherwise unknowable and makes producers cautious about everything.

Implementation patterns

• The contract is on a deliberately-published dataset, not on an internal table. The producer emits a designed event or table as its interface, and its internal schema stays internal — the same reasoning as an outbox event rather than raw change data capture.
• Schema with types and nullability, machine-readable and enforced at publication so a violation fails at the source rather than downstream.
• Semantics documented: what the field means, its units, its timezone, what null indicates, which states are possible. Semantic misunderstanding causes more incidents than schema breakage and is what a schema alone cannot express.
• Quality guarantees as measurable expectations — freshness, completeness, uniqueness of the key, accepted value ranges — monitored and alerted on, so a violation is detected by the producer rather than reported by the consumer.
• A change policy: additive changes freely, breaking changes with a stated notice period and a migration path. Additive-only evolution is what keeps this affordable, exactly as with API versioning.
• A registry of consumers, which is what makes the notice period actionable.
• Versioning, with the old version maintained through the notice period.
• Ownership on both sides, named.

Industry example

Data contracts became a mainstream practice as data mesh ideas spread, and they are the operational answer to the recurring complaint that analytical pipelines break constantly for reasons nobody upstream intended. The same reasoning underlies schema registries in streaming platforms, where compatibility rules are enforced at publication rather than discovered at consumption.

The structural insight is shared with API versioning and the outbox pattern: a public interface must be a deliberate artefact, distinct from the internal representation, because anything a consumer can reach becomes a contract whether or not anybody intended it to.

Failure scenarios

• Contracts on internal tables, which merely formalises the coupling rather than removing it.
• Schema without semantics, so a technically-valid change alters the meaning and every consumer is quietly wrong.
• Quality expectations documented but not monitored, so violations are found by consumers.
• No consumer registry, making the notice period meaningless.
• A change policy nobody enforces, after which the contract is a document.
• Contracts on everything, which is unaffordable — they belong on the datasets that cross team boundaries and matter, not on every table.
• The producer bearing all the cost and receiving none of the benefit, which is why the "you may change anything not in the contract" clause is essential to adoption.

Trade-offs

Contracts impose real cost on producers: publishing a deliberate dataset separate from their internal schema, maintaining it, monitoring quality, and honouring notice periods for changes they would otherwise make freely.

They also slow down the analytical side, which currently helps itself to whatever it can reach. A contract means requesting a field rather than joining to a table, and where the producer is unresponsive this is genuinely worse for the consumer in the short term.

The trade is producer effort and consumer convenience in exchange for a dependency that both parties can see and manage. For a small organisation with a handful of pipelines, direct reads and good communication are proportionate. At the scale where nobody can enumerate the consumers of a table, the implicit version has already failed — the breakages are simply being absorbed as background noise.

Interview question

"A backend team renamed a column and broke six pipelines and a board report. Tell me why that was not their fault, what you would put in place, and what you would offer them in return so that they agree to it."