A million devices report telemetry every ten seconds. How should ingest, buffering, backpressure and storage be designed, and where do naive designs fail?

The arithmetic first

A million devices at one message per ten seconds is 100,000 messages per second, sustained, with no natural quiet period. The characteristics that matter:

• Small messages at very high rate, so per-message overhead dominates — protocol handshakes, connection setup, authentication and serialisation cost more than the payload.
• Write-heavy with almost no reads on the ingest path.
• Devices with intermittent connectivity, which buffer and then burst.
• A long tail of misbehaving devices — a firmware bug that reports every second instead of every ten, a device stuck in a reconnection loop, a clock set to 1970.

The design

• A protocol suited to the shape: MQTT or similar with persistent connections and small framing, rather than a fresh HTTPS request per message. Connection establishment and TLS handshakes at 100,000 per second are the dominant cost in a naive design, and persistent connections remove them.
• Edge or gateway aggregation where the topology permits, so many devices report through one connection and the ingest tier sees far fewer.
• Batching on the device, sending ten readings every hundred seconds rather than one every ten — which reduces the message rate by an order of magnitude and trades a small amount of freshness for it. This is usually the highest-value change available.
• A durable log at the ingest boundary, decoupling acceptance from processing so a slow consumer does not reject device data.
• Partitioning by device identifier, giving per-device ordering and horizontal parallelism.
• A time-series or columnar store for the retained data, with downsampling: full resolution for days, aggregates for months, since almost all queries are recent and the retention cost is otherwise unbounded.

Backpressure and misbehaviour

• Per-device rate limits enforced at the edge, because a firmware bug in a fleet of a million devices is a distributed denial of service against your own platform, and it will happen.
• Reject cheaply, so a rejection costs far less than acceptance — otherwise the limiter is the bottleneck.
• Server-directed backoff, telling devices when to return, since client-side backoff cannot be updated on devices already in the field and that is exactly where the problem originates.
• Jittered reconnection, mandatory, or a platform blip produces a synchronised reconnection storm from a million devices — the recovery load being an order of magnitude above steady state, which is the failure that turns a short outage into a long one.
• Bounded per-device buffering on the device, with an eviction policy chosen deliberately.
• Quarantine for misbehaving devices, so one bad firmware cohort can be isolated without affecting the rest.

Where naive designs fail

• A request per message over HTTPS, dominated by handshake cost.
• Synchronous writes to a database on the ingest path, coupling device acceptance to database availability.
• No per-device limits, so a firmware bug saturates the platform.
• Unjittered reconnection, producing a self-sustaining storm after any blip.
• Trusting device timestamps, where clock skew and unset clocks produce data far outside the expected window — so both device time and server receipt time must be recorded, and the ingest must handle a device claiming 1970.
• Full-resolution retention forever, which is an unbounded and rapidly dominant cost.
• Ingest capacity planned for average rather than for the reconnection burst, which is the number that actually matters.