Conflation

When updates to an entity supersede one another — a price, a vehicle position, a device status — a consumer that has fallen behind gains nothing from the intermediate values. Conflation keeps only the latest value per key and discards the rest, so the queue depth is bounded by the number of distinct entities rather than by the message rate.

Why it matters

It is the correct backpressure response for a large class of high-volume streams, and it is structurally different from buffering or shedding: nothing the consumer needs is lost, because the value it would have read has already been replaced.

It also converts an unbounded problem into a bounded one. A fleet of a hundred thousand vehicles produces an unbounded message stream and a bounded state — conflation makes the system's memory proportional to the second.

Implementation patterns

• A last-value map keyed by entity, with the consumer reading the current value rather than a queue.
• Apply it only where updates genuinely supersede. Positions, prices and status flags conflate. Orders, transactions and state transitions do not — discarding one of those is data loss, and the distinction has to be explicit in the design rather than a property of the transport.
• Separate conflatable and non-conflatable streams onto different transports. They have opposite requirements and cannot share a channel: one must drop under pressure and the other must never drop.
• Publish the conflation ratio as a metric, because it measures how far behind consumers are running and it degrades gracefully rather than alerting suddenly.
• Combine with edge filtering. A device that sends only on meaningful change — moved beyond a threshold, changed heading, changed state, plus a low-frequency heartbeat — removes most of the volume before it ever reaches the pipeline, which is cheaper than conflating it later.

Industry example

A trading platform such as Zerodha must conflate market data during a burst: a client that cannot keep up needs the current price, not a queue of stale ones. Crucially, conflation is available to market data and categorically unavailable to order flow, which is precisely why the two cannot share a transport — the correct overload behaviour for one is the worst possible behaviour for the other.

A logistics platform such as Delhivery faces the same split from the other direction: live map tracking should conflate to the latest position, while route reconstruction needs every point but tolerates minutes of delay. Forcing both through one pipeline means paying the strictest requirement of each — completeness and low latency — which is the expensive combination.

Failure scenarios

• Conflating something that does not supersede, silently losing transactions or state transitions.
• One pipeline for both classes, so either the live path is slow or the durable path loses data.
• Conflation applied at the wrong layer — after the expensive processing rather than before it — so the cost is already paid.
• No visibility, so nobody knows consumers are conflating heavily until a user reports missing detail.

Trade-offs

Conflation loses history by design, which is correct for the live path and unacceptable for audit, analytics or replay. Any system that conflates therefore needs a second, durable path for the same data if that history has value — and running two paths costs more than one.

The alternative to conflation under pressure is buffering (which grows unboundedly), shedding (which loses the latest value as readily as the oldest) or slowing the producer (which is often physically impossible when the producer is a vehicle or an exchange). Against those, conflation is usually the best available option for the live path — it is the only one whose loss is guaranteed to be information you no longer need.

Interview question

"Your live tracking pipeline is falling behind and your audit pipeline is fine, both fed from the same topic. What do you change, and what do you have to be careful not to conflate?"