A logistics platform ingests location updates from a large vehicle fleet every few seconds. The ingestion queue grows steadily even though workers are healthy. How do you determine the cause, and what are the possible fixes?
Show the full answer Hide the answer
Distinguishing the causes
A growing queue with healthy workers has six plausible causes, and they need different fixes:
| Cause | Signal that identifies it |
|---|---|
| Insufficient capacity | Workers at high CPU, per-message latency normal |
| Slow downstream | Workers idle-but-blocked, time spent in downstream calls |
| Oversized messages | Throughput in messages falls while bytes stays flat |
| Poison message | One partition stalls, others fine; repeated redeliveries of the same key |
| Inefficient processing | Per-message CPU far above what the work justifies |
| Wrong workload model | Volume is inherent, and no amount of workers is economic |
The instinct is to add workers, which fixes exactly one of the six and makes two of them worse — more workers against a slow downstream deepens the downstream's queue and can tip it into failure.
The fix that usually applies to fleet telemetry
For location data at this volume the answer is almost always the last row: the workload model is wrong. Every vehicle sending a position every few seconds produces an enormous stream in which the vast majority of messages carry almost no information — a stationary vehicle reporting the same coordinates.
The architectural corrections, in order of leverage:
- Filter at the edge. The device sends on meaningful change — moved more than a threshold, changed heading, changed state — plus a low-frequency heartbeat. This is a very large reduction and it costs nothing downstream.
- Aggregate before the expensive stage. Downstream consumers usually need "where is this vehicle now" and "what route did it take", neither of which requires every point at full fidelity in real time.
- Separate the paths by consequence. Live tracking for a customer watching a map needs the latest position and can drop everything older. Route reconstruction needs completeness but not latency. These are different systems and forcing them through one pipeline means paying the strictest requirement of each.
- Conflate rather than queue for the live path. Keeping the latest value per vehicle and discarding superseded ones bounds the work by fleet size instead of by message rate — the queue cannot grow past the number of vehicles.
The principle
An unbounded queue is a design defect, not a buffer. It converts a throughput mismatch into a memory exhaustion and turns a degradation into an outage, while hiding the mismatch until the moment it becomes fatal. Bound the queue, and decide explicitly what happens when it is full: shed, sample, conflate, or reject upstream. Any of those is better than growing.