Message Formats
JSON, Protobuf, Avro and the rest — chosen by who the consumer is and how the schema will evolve.
Definition
The wire format determines payload size, parsing cost, and — most importantly — how schema change is governed.
| Format | Strengths | Weaknesses |
|---|---|---|
| JSON | Universal, human-readable, no tooling | Verbose, no schema by default, weak typing |
| Protobuf | Compact, fast, strong evolution rules, code generation | Binary, needs schema distribution |
| Avro | Compact, schema travels with data, excellent for data pipelines | Less convenient for RPC |
| MessagePack / CBOR | Compact JSON-like, minimal ceremony | No schema, no evolution rules |
The decision that actually matters
Not size or speed — schema governance. A published message is an API consumed by systems you may not be able to enumerate, so the question is what stops someone breaking it.
JSON with no schema means the contract exists in documentation and in the heads of consumers, and it breaks silently: a field's type changes from number to string and downstream code coerces it, producing wrong results rather than errors.
Protobuf and Avro both carry explicit compatibility rules that can be enforced in CI: a change that would break existing readers fails the build. That mechanical enforcement is the entire value.
Industry example
LinkedIn's use of a central event log made schema governance an operational necessity rather than a preference. Once producers publish once and unknown consumers subscribe, a producer team cannot enumerate who would break — so the only workable control is a schema registry with compatibility checks in the pipeline.
The registry does three things worth naming: it stores every version of every schema; it enforces a compatibility mode per subject (backward, forward, or full) at registration time; and it lets messages carry a small schema ID instead of the schema itself, which is what makes self-describing data affordable at high volume.
Without that, "we will be careful" is the policy, and it fails in the first year.
Compatibility modes, which are the operational core
- Backward compatible — new schema can read old data. Required when consumers upgrade first.
- Forward compatible — old schema can read new data. Required when producers upgrade first.
- Full — both. Required when you cannot control the order, which in practice is most of the time.
Choosing the mode is choosing your deployment freedom, and it should be a conscious decision per topic.
Failure scenarios
- A field's type changed in a schemaless format, coerced downstream into wrong values.
- A field removed that a consumer you did not know about depended on.
- Enum value added, and consumers with strict parsing reject the message.
- Schema embedded in every message at high volume, so encoding overhead exceeds the payload.
- Registry not enforced in CI, so it documents schemas rather than governing them.
Interview question
"A producer team wants to rename a field. What do you tell them, and what mechanism makes the answer enforceable rather than advisory?"