Clock Drift
Device clocks diverging from real time, which corrupts ordering, expires certificates early or late, and produces data that appears to arrive from the past.
A device's real-time clock drifts, loses time entirely when power is removed, and may start at an epoch default with no way to know it is wrong. In a fleet, some proportion is always significantly incorrect.
The consequences reach further than timestamps. Ordering breaks: telemetry timestamped by the device arrives apparently out of sequence, and any windowed aggregation over event time produces wrong results. TLS fails: certificate validity is checked against local time, so a device with a badly wrong clock cannot establish a connection — and cannot reach the time server it would use to correct itself, which is a genuine deadlock that has stalled real deployments. Scheduling drifts, so devices meant to report hourly diverge from each other and from the plan.
The mitigations are ordinary once the problem is anticipated. Record both device time and server receipt time, and treat server time as authoritative for ordering while retaining device time for local reasoning. Synchronise on every connection. Allow a bootstrap path that tolerates an invalid clock long enough to obtain the correct time. And monitor the distribution of device-to-server time offset across the fleet, because a sudden shift in it is an early indicator of a firmware or configuration fault.