A regional power cut takes out 9,000 building-controller devices for six hours. Power returns and 8,700 of them reconnect within ten minutes. The other 300 never come back, and the gateway logs show TLS handshake failures with "certificate is not yet valid". The certificates do not expire for another year. What failed, and which design decision made it possible?

The trigger

Those 300 devices have a real-time clock with a dead or absent backup cell, so losing mains power reset the clock to its epoch — 1 January 1970 on a Unix-derived stack, 1 January 2000 on some embedded ones. On boot the device opens a TLS connection, reads the server certificate, and checks notBefore against the only time it has. A certificate issued in 2024 is not yet valid in 1970, so the handshake is refused by the client, correctly, according to the rules it was given.

Why it propagates instead of self-healing

The device cannot fix its clock, because the way it learns the time is over the network it cannot open. Plain NTP would work, but hardened fleets commonly use NTS, DNS over TLS, or fetch time from an HTTPS endpoint — every one of which validates a certificate first. That is a circular dependency: correct time requires a trusted connection, and a trusted connection requires correct time.

The 8,700 devices that recovered are not better engineered. They are the ones whose RTC cell still holds charge, or that were power-cycled briefly enough that the clock survived. The population that fails is a function of battery age, so this failure arrives gradually and then all at once, five to eight years into a deployment.

Why detection lagged

Nothing errored on the platform side. The devices were simply absent, and absence looks like a site with no power or a customer who unplugged a panel. Fleet dashboards report connected counts, and a 3% shortfall after a regional incident reads as normal attrition. The signal that would have named it — a count of handshake failures by TLS alert code, keyed to certificate_not_yet_valid — is on the gateway, which most fleets do not aggregate by reason.

The structural fix, and the tempting local one

The tempting fix is to ship a firmware update that ignores certificate validity dates. That removes the expiry check from the whole fleet permanently to solve a clock problem, and it is how a revoked or long-expired server certificate later becomes usable by an attacker.

What actually works:

1. Make time acquisition possible without a valid clock. Allow one bootstrap path — plain NTP or a signed time response from a pinned key — that does not require date validation, then re-validate everything once the clock is set.
2. Persist monotonic time across reboots. Write the last known good time to flash every few minutes and boot from it. A device that resumes at "a few minutes before the outage" passes date validation and corrects itself on the first NTP exchange.
3. Fit the RTC cell as a serviceable part with a known life, and treat its exhaustion as a scheduled event rather than a surprise.
4. Alert on handshake failure reasons, not just on connection counts.

Common weak answers

• "Extend the certificate validity window to ten years." A certificate that is not yet valid in 1970 is still not yet valid in 1970. Longer certificates change nothing here and weaken you elsewhere.
• "The devices should retry until the clock is right." Retrying costs power and fixes nothing, because the clock is only corrected by a network exchange the device cannot complete.
• "Monitor the fleet more closely." Connected counts were already monitored. Choose a signal with a cause attached — handshake failures grouped by TLS alert reason — or the next occurrence looks identical.

The general lesson

Any device that can lose its notion of time has a dependency loop hidden inside its security stack, and the loop is invisible while the batteries are young. The same shape appears wherever trust establishment depends on state the device can lose: a certificate pinned to a hostname the device cannot resolve, a token whose expiry it cannot evaluate, a signature it cannot check without a root it has to fetch.