A producer team renames a column. The data contract is defined and the contract test is green. Three hours later four downstream models fail and a board dashboard is blank. The test lived in the consumer's repository and ran on the consumer's schedule. What is the structural defect?

The trigger

A rename shipped in the producer's release. Nothing in the producer's pipeline knew a contract existed, because the artefact that encoded it was in a different repository, owned by a different team, and executed after the change had already landed.

Why it propagated

A test that runs downstream of a change detects it; it does not prevent it. By the time the consumer's scheduled run went red, the new column name was in production, the old one was gone, and the fix was a producer deployment rather than a merge-request rejection. The three-hour gap is the schedule, and it is also the window in which the blank dashboard was seen by the people the dashboard exists for.

The second propagation path is worse: the consumer's failure looks like a consumer problem. The page goes to the analytics team, who spend the first half-hour looking at their own code, because nothing in the alert says "an upstream contract was violated at 09:14 by commit abc123".

Why detection lagged

Two schedules and no coupling between them. The producer deploys continuously, the consumer's tests run hourly, and the contract — the thing that was supposed to bind them — was a file that neither pipeline treated as a gate.

The structural fix

Move the enforcement point to the producer's merge request. The contract becomes a check the producer must pass to ship: schema compatibility evaluated against the published contract, failing the build on a breaking change. That is the only placement where "prevented" is possible rather than "detected".

Concretely:

1. The contract is published by the producer and versioned alongside their code, so a change to the data and a change to the contract are one commit.
2. Compatibility rules are explicit: adding an optional column is compatible, renaming or removing one is not, tightening a type is not.
3. A breaking change is allowed, but only through a path: publish a new version, run both, notify consumers, remove the old after a stated window. The contract's job is not to prevent change; it is to make breaking change deliberate and observable.
4. The consumer's test stays, downgraded to what it should always have been: a canary against reality rather than the primary control.

The tempting local fix

Running the consumer's test more often. It narrows the window and changes nothing structural, and it is the fix most often shipped because it is the one the consumer team can do alone. That is the real lesson: the defect is organisational and the available fix is technical, so the technical one gets applied for years.

The general lesson, and when not to apply it

A contract is enforced where it can block, and nowhere else. The same reasoning decides where masking policies live, where schema checks run and where approval gates sit: ask which side of the boundary can stop the action, and put the control there. If the answer is "the side that does not own the artefact", the contract is documentation.

Choose the producer-side gate whenever the producer's pipeline can run the check at all, which is almost always; the exception is a source outside your organisation, where detection is genuinely the only option and the honest design puts a quarantine step between the feed and anything that depends on it. What the gate costs the producer is real and should be stated when proposing it: a check on every merge, roughly 30 seconds of build time, and an occasional blocked release that previously would have shipped. That is the price of moving the breakage from four downstream teams to one merge request, and it is worth naming rather than minimising.