A producer adds one optional field to an event schema and deploys. The registry accepts the new version. Within minutes, consumers that have not been redeployed start failing to deserialise every message on the topic. What was wrong with "optional"?

The first three things to check

1. The field's default value in the new schema. This is almost always the answer.
2. The registry's compatibility mode for that subject - what it checks, and in which direction.
3. Which schema version the failing consumers hold, because the failure is about the reader's schema and not the writer's.

The diagnosis

In Avro, "optional" is not a property a field has on its own. A reader can only skip a field it does not know about if the schema it is reading with supplies a default for it. When a consumer holding version 1 reads data written with version 2, resolution walks the reader's schema: for each field the reader expects it looks in the writer's data, and for each field the writer has that the reader does not, it skips. The problem is the reverse direction and the one people get wrong - a field present in the data but absent from the reader's schema is fine; a field the reader expects and cannot find needs a default, or resolution fails.

A nullable union ["null", "string"] without "default": null declares a field that may hold null, not a field that may be absent. Those are different statements, and only the default makes the schema evolvable.

The registry accepted it because compatibility was checked in the direction that was configured, which is where the second check comes in: Confluent's Schema Registry, released in 2015, defaults to BACKWARD - which asks whether a consumer using the new schema can read data written with the old one. That is satisfied here. The failing direction - old consumer, new data - is FORWARD, and checking both requires FULL. The decision rule: match the compatibility mode to your deployment order, or choose FULL and stop depending on order at all. FULL costs the freedom to make one-directional changes, which is a price worth paying once more than about 10 teams share the topic.

The misleading clue

"The registry accepted it." Teams read registry acceptance as a guarantee of safety, when it is a guarantee about one specific relation between two specific versions in one direction. The registry enforces the policy you configured, and most estates configure the one that matches the deployment order they do not actually follow.

The fix

1. Give every added field a default. For a nullable field that means both the union and "default": null.
2. Set the subject's compatibility mode to FULL where producers and consumers deploy independently, which is the whole point of having a registry in an estate with many teams.
3. Decide and document the deployment order - consumers first for backward compatibility, producers first for forward - and make the compatibility mode match it rather than hoping.
4. Never remove a required field or change a type in place. Add the new field, migrate readers, then retire the old one - the same expand-and-contract discipline a database column needs.
5. Put a consumer-reads-new-data case in CI, with the actual previous schema, because that is the check the registry was not performing.

When this is the wrong answer

If the failure affects only consumers that were redeployed, the direction is reversed and the cause is the opposite omission - typically a removed field or a narrowed type. And if the deserialisation error names a schema id that is not in the registry at all, this is not a compatibility problem but a topic receiving data from a producer pointed at a different registry - check the schema id before the schema.