Schema Evolution and Compatibility Modes

Schemas change. The question a registry answers is whether a proposed change can be deployed without breaking anything, and the answer depends on a direction that people routinely get backwards. Getting it right is the difference between a safe rollout and an outage caused by a change everyone agreed was additive.

The three modes

Backward compatible means a consumer using the new schema can read data written with the old schema. Adding an optional field with a default is backward compatible: the new reader encounters old data lacking the field and substitutes the default. Deleting a field is also backward compatible, since the new reader simply does not look for it.

Forward compatible means a consumer using the old schema can read data written with the new schema. Adding a field is forward compatible when the old reader ignores unknown fields. Deleting a required field is not, since the old reader needs it.

Full compatibility requires both, which restricts changes to adding and removing optional fields with defaults.

Which one you need depends on deployment order

This is where the confusion lives, and the rule is mechanical.

If consumers upgrade first, they will read old data written by not-yet-upgraded producers, so you need backward compatibility.

If producers upgrade first, old consumers will read new data, so you need forward compatibility.

If you cannot control the order, which is the normal situation with many independent consumers, you need full compatibility.

For an event log with retained history the requirement is stronger still: a new consumer replaying from the beginning must read every version ever written, so backward compatibility must hold transitively against all previous schemas, not just the immediately preceding one. Registries expose this as a transitive mode and it is the correct setting for anything replayable.

Always breaking, regardless of mode

Renaming a field is a delete plus an add. Narrowing a type, such as long to int, breaks on values that no longer fit. Changing units, currency or timezone keeps the schema valid and breaks every consumer semantically. Adding a required field with no default breaks every old writer. Changing the meaning of an enum value is invisible to any automated check and is the most dangerous of the set.

Field-identifier-based formats such as Protobuf tolerate renaming because identity is the tag number rather than the name, which is one reason they are preferred for long-lived contracts. Name-based formats such as JSON Schema and Avro treat a rename as a structural change.

When it breaks

Optional with a default is not the same as nullable. A default lets an old reader materialise a value; nullability lets the value be absent. Confusing them produces schemas that pass compatibility checks and fail at read time, particularly across language boundaries where absent and null are represented differently.

Compatibility checks do not cover semantics. Every check here is structural. The failure mode that costs the most, a field whose meaning changed, passes cleanly, which is why distribution monitoring is a necessary complement rather than an optional extra.

Columnar files and streams evolve differently. A table format tracks columns by stable identifier and can rename without rewriting data. A stream's compatibility is governed by its registry and by consumers already deployed. The same logical change can be safe in one and breaking in the other, so a schema change needs evaluating per medium.

Defaults are frequently wrong. A numeric field defaulting to zero for old records is indistinguishable from a genuine zero, which corrupts every aggregate over the transition period. Nullable with an explicit absent representation is usually more honest, at the cost of every consumer handling it.