Device Telemetry at Scale
The volume problem created by many devices each emitting small frequent messages, where the answer is filtering at the edge rather than ingesting everything.
The arithmetic is what makes this a distinct problem. A hundred thousand devices reporting every ten seconds is 864 million messages a day. Each is a few hundred bytes, so the volume is modest; the message rate is what strains ingestion, and the retention is what strains the budget.
The instinct to capture everything at full resolution forever is expensive and usually wrong, because the value of a temperature reading from eighteen months ago at one-second granularity is close to zero while its storage cost is not.
The design responses, in order of effectiveness. Filter at the edge: report on change beyond a threshold rather than on a timer, which typically eliminates the large majority of messages from sensors whose readings are stable. Aggregate at the gateway: send a minute's statistics rather than sixty readings, keeping the shape and discarding the redundancy. Tier the retention: full resolution for days, downsampled aggregates for months, coarse aggregates for years. And batch transmission, since per-message overhead dominates at these sizes.
The exception that must be designed for separately: anomalies and events need full resolution and immediate transmission, so the filtering must be conditional rather than uniform. A system that downsamples the one minute in which a machine failed has optimised away the only data anyone wanted.