An order service publishes events consumed by six teams. Every consumer immediately calls back for order details. What is wrong and what do you change?
Show the full answer Hide the answer
The failure
Thin notification events ("order 123 changed") produce a callback stampede: every event triggers six synchronous calls back to the producer, and the producer's availability becomes every consumer's availability. The asynchrony is illusory.
The change
Event-carried state transfer — carry the data consumers need. They act without calling back, and keep working when the producer is down. That is the actual benefit of events, and this system was not getting it.
But not the whole internal model. Publishing the producer's internal representation couples every consumer to a schema never designed as a contract, and the producer can no longer refactor freely.
The principle: carry what the common consumers need, not everything. If most consumers immediately fetch the same three fields, those fields belong in the event. If one consumer needs a large rarely-used attachment, it can fetch that.
Determine this from evidence — look at what the callbacks actually request.
Design the event as a published contract
A deliberate schema, versioned and owned, distinct from the internal model. This is the discipline that makes the difference between an event stream that lasts and one that becomes a liability.
A registry with a compatibility mode enforced in CI. For a retained stream, full transitive compatibility is the conservative and usually correct setting — a sequence of individually-safe changes can collectively break a replay from the beginning.
Understand what each mode implies about upgrade order: backward means upgrade consumers first, forward means producers first, full means order does not matter.
Include: event ID (for consumer deduplication, since delivery is at-least-once), event type, schema version, timestamp, aggregate ID, and a correlation ID so a flow can be traced across services.
Why versioning is harder here than for APIs
There is no synchronous caller to negotiate with, and events may be replayed from a retained log months later. A breaking change is discovered by a consumer failing on old data, not by an integration test.
Rules: add optional fields; never remove, rename, narrow a type, or change the meaning of an existing field. For a genuinely breaking change, publish a new event type alongside the old for a transition period, or upcast on read — which is a permanent maintenance obligation.
What a strong answer adds
Noting the ordering constraint: ordering is per partition per topic, so OrderPlaced, OrderShipped
and OrderCancelled usually need to share a topic. That in turn rules out TopicName subject naming and
points to TopicRecordName — a schema-registry decision that follows directly from a domain
requirement.