A platform's event streams have many producers and consumers across teams. Someone deploys a change that removes a field, and three downstream consumers fail overnight. What should have prevented this?
Show the full answer Hide the answer
What should have prevented it
A schema registry enforcing compatibility as a build gate, not as documentation.
The mechanism: every schema is registered with a compatibility mode. Before a producer can publish a new version, the registry checks the change against the rule. Removing a field violates backward compatibility, so the registration is rejected — and because registration happens in CI, the producer's build fails before merge.
The essential property is that the check happens where the change is made, by the person making it, before it can reach production. A registry consulted at runtime, or a wiki page describing the rules, prevents nothing.
The compatibility modes and what they mean
- Backward compatible — new schema can read data written with the old one. Required when consumers upgrade after producers. Allows adding optional fields; forbids removing fields or making optional fields required.
- Forward compatible — old schema can read data written with the new one. Required when consumers upgrade before producers, or when old consumers must keep working against new data.
- Full — both. The safest and most restrictive.
For a shared event stream with independently-deploying consumers, full compatibility is usually the right default, because you cannot control the upgrade order and both directions occur.
Why the failure surfaced overnight
Worth noting, because it is characteristic. The consumers that failed were probably batch jobs. Interactive consumers would have failed immediately and loudly; batch consumers fail at 2 a.m. and are discovered the next morning, having skipped a night of data. The delay between the change and the visible consequence is what makes this class of failure expensive — the causal link is not obvious.
What else the registry provides
Discoverability. Which schemas exist, who produces them, what they contain. Without this, the only way to know what an event stream carries is to read producer code.
Consumer registration, so a producer can see who depends on a schema before changing it. Most fear of changing an event is fear of unknown consumers.
Versioned history, so a consumer can interpret old data. This matters more than teams expect: events are immutable and retained, so every historical schema version must remain interpretable for as long as the data is retained. Replay code accumulates handling for versions written by systems nobody remembers.
The organisational point
The registry converts a social agreement into a mechanical constraint. Teams agreeing not to break consumers works until a deadline; a build that fails works always.
And the deeper framing: once an event has external consumers, its schema is a public API. It deserves the same versioning discipline, deprecation process and consumer measurement as any published interface — which is exactly what most organisations fail to apply, because an event does not look like an API.