A telemetry platform ingests billions of events daily from customer agents. Which message format should the wire protocol use, and what does the choice affect beyond size?
Show the full answer Hide the answer
What the choice affects
Bandwidth and cost. At billions of events daily, the difference between a verbose text encoding and a compact binary one is a direct, large infrastructure cost — paid by the customer's egress as well as the platform's ingress.
CPU on both sides. Parsing is not free. On the agent, running on the customer's production hosts, CPU consumption is a competitive concern: an agent that noticeably loads customer machines gets uninstalled. On the platform, parsing at this volume is a substantial share of ingest cost.
Schema evolution, which is the underrated one. Agents are deployed on customer infrastructure and upgrade slowly — some run versions years old. The platform must accept events from every version simultaneously, forever. A format with explicit field numbering and defined compatibility rules makes this tractable; a format without one makes every schema change a coordination problem with an unreachable population.
Compression behaviour. Binary formats with repeated structure compress well in batches; the practical choice is a compact encoding and batch compression, because per-message compression wastes most of the benefit.
Why the alternatives lose
JSON. Human-readable, universally supported, and expensive on every axis that matters here: larger on the wire, slower to parse, no schema enforcement, and no defined evolution rules. Excellent for a low-volume public API; wrong for a high-volume agent protocol.
A custom binary format. Tempting, and it forgoes the ecosystem: no generated clients across languages, no tooling, no schema registry integration, and every compatibility rule invented and maintained by you. The marginal gain over a mature schema-defined format is small; the maintenance cost is permanent.
XML. Verbose, slow to parse, and its schema validation is not the kind of compatibility guarantee this problem needs.
The rule that matters most in practice
Field numbers are permanent. Removing a field and later reusing its number produces silent data corruption — old agents send one meaning, the platform reads another, and nothing errors. This is why schema changes deserve genuine review, and why the registry must enforce compatibility rather than trusting discipline.
The pragmatic combination
Compact binary schema-defined format for the high-volume agent protocol; JSON for the low-volume public API where developer experience dominates and the volume makes efficiency irrelevant.
Format choice follows volume and audience, and most platforms correctly run both — the mistake is picking one for the whole system on ideological grounds.