Design ingestion for millions of devices sending telemetry continuously over unreliable networks.

The shape of the problem

Very high connection count, small messages, unreliable and intermittent connectivity, constrained devices, and a strong daily pattern. The bottleneck is connections and message rate, not bytes.

The ingestion tier

• A protocol suited to constrained devices and poor networks — persistent lightweight sessions rather than a request per reading, which avoids paying connection setup repeatedly.
• A connection tier separate from the processing tier, so processing slowness does not drop connections and reconnection storms do not overwhelm processing.
• Authentication per device with individual revocation, checked at connect.
• Backpressure that sheds rather than queues indefinitely — telemetry is usually more valuable fresh than complete, and an unbounded queue converts a slowdown into a memory-exhaustion failure.
• Partitioning by device identifier, giving per-device ordering with parallelism.

The device-side design

• Batch and compress on the device, since radio wake-ups dominate power consumption.
• Send current state, not the buffered backlog, after an outage. Old readings are usually worthless and the aggregate flush is a large useless burst.
• Exponential backoff with jitter on reconnection, without which a network event causes a synchronised reconnect storm — the most common way an IoT platform takes itself down after an unrelated outage.
• Bounded local buffering with a defined overflow policy.

What must be planned before launch

Retention and rollup, decided at ingestion. Raw high-frequency telemetry is enormous and mostly uninteresting after a short window; rollups are far cheaper to compute at ingestion than to backfill, and raw data never retained cannot be rolled up differently later.

And a fleet-scale reconnection plan, because the recovery path carries several times normal load and is usually the path nobody load-tested.