IoT Ingest Architecture

Definition

The path from a device emitting telemetry to that data being durably stored and usable.

The characteristics that shape it

• Very high connection count, frequently exceeding message rate in difficulty. Millions of devices each sending occasionally is a connection management problem, not a throughput one.
• Small, frequent messages, so per-message overhead dominates and protocol choice matters — a lightweight publish-subscribe protocol over a persistent connection beats an HTTP request per reading.
• Bursty and correlated arrival. Devices reconnect together after a network event, producing a reconnection storm that is a classic self-inflicted second outage.
• Unreliable clients, sending malformed data, duplicating, and reporting bad timestamps.
• Ordering that cannot be assumed, and event time that differs substantially from arrival time because devices buffer while offline.

The architecture

Ingest gateway terminating device connections, authenticating per device, and validating early. This tier scales on connections, not on requests, and its capacity is measured in concurrent connections and memory per connection.

A durable log immediately. Accept, persist, acknowledge — then process asynchronously. This decouples ingest availability from processing capacity, which is what allows the pipeline to fall behind without dropping data.

Backpressure and shedding, defined in advance. When processing cannot keep up: buffer boundedly, then shed by priority. Devices should be told to back off rather than being silently dropped, or they retry immediately and make it worse.

Time handling. Record both device time and arrival time; treat device clocks as unreliable; process on event time with an explicit lateness policy, since a device offline for a day will deliver a day of backlog at once.

Deduplication, because at-least-once delivery plus device retries guarantees duplicates.

Failure scenarios

• Reconnection storms after a network event, overwhelming the gateway. Jittered backoff on the device is the primary mitigation and must be in the firmware.
• Synchronous processing at ingest, so a slow downstream drops device data.
• Device timestamps trusted, producing data ordered by a clock that is wrong.
• Unbounded buffering, converting a throughput problem into an outage.
• No per-device rate limit, so one malfunctioning device floods the pipeline.

Interview question

"Two million devices reconnect simultaneously after a network partition. What happens and what prevents it?"