A producer adds a new value, HELD_FOR_REVIEW, to the status enum of a widely consumed event. The schema registry's backward-compatibility check passes and the change deploys. Within an hour two of eleven consumers are dead-lettering every message and one is quietly booking the wrong outcome. What happened?

Second by second, what happens

The producer deploys. The first order enters review and emits status: HELD_FOR_REVIEW. Eleven consumers receive it and divide into three behaviours within one poll cycle.

Consumers whose generated types close the enum reject the record at deserialisation. A generated Java or C# enum has a fixed set of members; an unknown symbol throws before any business code runs. Those consumers dead-letter everything after that offset, including the ordinary messages interleaved behind it, so a change affecting 0.3% of orders stops 100% of the stream. The blast radius is the partition, not the affected records.

Consumers reading with an open enum keep going. Protobuf's wire format carries enums as varints and a reader that does not recognise the number preserves it as an unknown value rather than failing, so these consumers survive. Survival is not the same as correctness.

One consumer has an else branch. Its status mapping is a switch with a default that treats anything unrecognised as CONFIRMED, because when it was written every status was either confirmed or cancelled. It books held orders as confirmed. Nothing errors, nothing is dead-lettered, and the mistake surfaces days later in reconciliation.

Why the compatibility check passed

Registry compatibility modes reason about the schema, not about the consumers' generated code. Adding a symbol to an enum is a legal, backward-compatible schema change: an old schema's data still reads under the new schema. The direction that breaks is the other one — new data read by an old reader — and that is what a forward-compatibility check covers. A registry set to BACKWARD is answering a question you did not ask. Setting it to FULL would have caught this, at the cost of forbidding other changes you do want.

What stops it

• Treat the set of enum values as part of the contract and set the compatibility mode to FULL for any topic whose consumers are not deployed in lockstep. That is a deployment-order decision, not a style preference.
• Publish the new value before producing it. Register the schema, give consumers a release window to handle it, verify with the registry which consumer versions are registered, then start emitting. Weeks, not minutes, when consumers belong to other teams.
• Model the open case explicitly. Reserve UNKNOWN = 0 and require consumers to map unrecognised values to it, then make UNKNOWN fail closed in business logic: hold the order, raise an alert, never assume the benign branch. A default branch that picks a real outcome is the dangerous pattern here, and a lint rule banning default: in status switches catches it in review.
• Alert on the signal that fires first: unknown-symbol deserialisation errors and dead-letter rate per consumer group, not consumer lag, which only moves once the damage is done.

When not to pay for this

If every consumer is in one repository and deploys together, an enum is fine and a string is worse, because the type system catches the missing case at compile time. The discipline above is the price of consumers you do not deploy. And if statuses are genuinely open-ended, as with carrier or provider codes, stop using an enum: carry a string plus a documented code list with an owner, and accept that consumers must handle values they have never seen.