Conflatable Classification

High-volume device telemetry contains two fundamentally different kinds of message, and they are frequently carried on the same transport.

Superseding messages — a position, a speed, a battery level, a status — where a newer value replaces an older one and the intermediate values have no independent worth.

Discrete events — a fault code, a delivery confirmation, a trip completion, an update acknowledgement — where each one matters and dropping any is data loss.

Conflation is the correct overload behaviour for the first and unacceptable for the second, which is why they cannot share a path.

Why it matters

The classification determines the entire pipeline design. Conflating superseding data bounds work by device count rather than by message rate, converting an unbounded problem into a bounded one. Applying the same behaviour to discrete events loses information silently.

And the mistake is easy: both arrive from the same device over the same connection, and a transport that treats them identically is the simplest thing to build.

Implementation patterns

• Classify each stream explicitly in the design, not as a property of whichever transport was chosen.
• Separate transports or separate topics, with different overload behaviour and different delivery guarantees.
• Quality of service per message type — fire-and-forget for a position, acknowledged for a fault code — since one level for everything either wastes bandwidth or loses important events.
• Filter superseding data at the device, transmitting on meaningful change plus a heartbeat. The largest available reduction, at the cheapest possible point.
• Never filter discrete events at the device, and buffer them locally through connectivity gaps with idempotent submission on reconnection.
• Monitor the conflation ratio, which measures how far behind consumers are running and degrades gracefully rather than alerting suddenly.

Industry example

Logistics and mobility fleets — Delhivery, Porter, Ather — carry both kinds continuously. The characteristic failure is a pipeline built for the position stream that also carries fault codes, at which point an overload event drops the faults, and the loss is discovered when a vehicle problem was reported and never recorded.

Failure scenarios

• One transport for both, so overload behaviour is wrong for one of them.
• Conflating discrete events, silently losing them.
• Never conflating, so an unbounded queue turns a throughput mismatch into memory exhaustion.
• Device-side filtering applied to discrete events.
• Conflation ratio unmonitored, so consumers falling behind is invisible until detail is missing.

Trade-offs

Two paths cost more than one: two transports, two sets of monitoring, two failure modes, and a decision required for every new message type.

The alternative is a single path that is wrong for one of the two classes — either paying reliable-delivery cost for position updates that are worthless a second later, or accepting silent loss of the events that matter. Neither is acceptable at scale, which is why the classification is worth the operational cost.

Interview question

"Your telemetry pipeline is overloaded and starts dropping messages. Tell me which ones it should drop, which it must not, and what in your architecture makes that distinction."