IoT Ingest Architecture
Millions of small, unreliable, frequently duplicated messages arriving continuously.
5 to work through
-
advanced
A million devices report telemetry every ten seconds. How should ingest, buffering, backpressure and storage be designed, and where do naive designs fail?
3 min answer -
advanced
After a five-minute network outage at your ingest tier, 200,000 devices reconnect simultaneously and the authentication service collapses. How do you fix this?
2 min answer -
advanced
Design ingestion for millions of devices sending telemetry continuously over unreliable networks.
1 min answer -
advanced
Two million devices reconnect simultaneously after a network partition. What happens and what prevents it?
2 min answer -
advanced
WeChat's DAGOR overload control (SoCC 2018) sheds load using the time requests spend queued inside a service rather than response time, and drops by business and user priority. How would you apply that to a device-fleet ingest tier, and what must the device do for it to work?
3 min answer
2 terms in this topic
Ingest Fan-In
The shape of a system receiving many small messages from very many senders, where connection count rather than data volume is the primary constraint.
patternIoT Ingest Architecture
Receiving high-volume telemetry from a large device fleet — where connection count, backpressure and reconnection storms dominate the design.
Neighbouring topics
Edge, Mobile & IoT
General material on architecture beyond the data centre boundary.
Mobile App Architecture
Layering, navigation, background execution, and the platform rules you do not set.
Offline-First
Treating connectivity as an optimisation, with a local store as the source of truth.
Sync & Conflict Resolution
Two devices that both changed the same record while neither could see the other.
CRDTs
Data types that converge without coordination, and the semantics you must accept.
Mobile Release Strategy
Store review, staged rollout, and supporting versions you can never force off.
Push & Background Work
Delivery that is best-effort, and an operating system that will kill your process.
Device Identity
Identifying a thing rather than a person, and rotating a credential you cannot type.
Edge Compute Topologies
Regional, metro, on-premises and on-device, and what each tier is genuinely for.
Edge Functions
Short-lived compute at the CDN, its runtime limits, and what must stay at origin.
Edge Data Consistency
Replicated read state at hundreds of locations, and writes that still go to one.
Device Provisioning
Getting identity and configuration onto hardware at manufacture or first boot.
Fleet Management
Inventory, health, configuration and grouping across devices you will never see.
OTA Updates
Updating firmware over a flaky link, with rollback, and without bricking the device.
Constrained Protocols
MQTT, CoAP and their kin, chosen for power, packet size and intermittent links.
Device Telemetry at Scale
Deciding what a device sends, how often, and what is aggregated before it leaves.
Digital Twin
A server-side model of a physical thing's reported and desired state.
Edge Security & Attestation
Secure boot, hardware roots of trust, and proving what is running on a device.
Physical-World Failure Modes
Power loss, tampering, clock drift, thermal limits, and a truck through the fibre.