Time cannot go backwards, and other lies production believed
How production systems survive machine clocks that jump, drift and lie: the wall/monotonic split, smeared leap seconds, leases with fencing tokens, and bounded-uncertainty ordering.
Two postmortems (Cloudflare 2017, Azure leap day 2012), three Linux kernel commits, the Go and Rust standard-library design records, Kubernetes leader election, and the clock architectures of Google, Meta, AWS and CockroachDB, reconstructed into a reference clock stack, four decision records with flip conditions, a three-class failure catalogue and a build ladder. After reading, an architect can audit a codebase for wall-clock subtraction, decide when a lease needs a fencing token, and price commit-wait against uncertainty restarts on their own hardware.
The industry's fix for the leap second is to make every clock deliberately wrong, and no two operators make them wrong the same way: Google smears 24 hours noon to noon, Meta 17 hours from midnight, so two perfectly synced fleets can disagree by most of a second during a leap event.
What you get out of it
- The Go monotonic-clock gap was on file for 15 months (issue #12914, Oct 2015) and became an accepted design 25 days after Cloudflare's leap-second outage; the proposal rejects two-API designs because clock resets make misuse fail on every machine simultaneously, defeating redundancy.
- Rust walked the same road in reverse: it forced monotonicity in std in 2019 because platform clocks were untrustworthy, then removed the enforcement in 2022 when its worst-case cost measured above 100x, choosing saturating arithmetic as the survivable poison.
- The layer that implements time is not exempt from time bugs: the 2012 leap second deadlocked the Linux timekeeping path and fired CLOCK_REALTIME timers a second early, and the mainline commit message preserves the industry's mid-incident workaround, date -s "`date`".
- Kubernetes leader election ships with its clock caveat in the package comment: no fencing guarantee; exclusivity that matters must be enforced by the resource via a monotonically increasing token, not by the leaseholder's own good behaviour.
- Bounded uncertainty is purchasable: Spanner's measured epsilon averaged 4 ms on GPS-and-atomic hardware, making a ~5 ms commit wait affordable, while CockroachDB on 500 ms commodity bounds restarts transactions instead; cloud microsecond sync now moves that flip condition.
Scope
Why this, now. Cloud providers now rent microsecond-class clock error bounds (AWS Time Sync plus ClockBound), which reopens the wait-versus-retry ordering decision most teams last made on 500 ms NTP assumptions.
What it does not cover. NTP and PTP protocol engineering and security, GPS spoofing and resilience, timezone and tzdata handling, and regulatory clock-sync mandates in finance.
Other field guides
Rent the store, own the layer in front of it: ten years of Figma
A single-company dig reconstructed from Figma's own postmortems and design posts, its S-1, and first-hand git archaeology of Redis, PgBouncer and Ske…
38 sources · 18 organisations · 3 postmortemsThe workspace stopped being the shard: ten years of Slack
A single-company dig reconstructed from Slack's own postmortems, engineering posts and QCon talks, and from first-hand git archaeology of the Vitess …
32 sources · 12 organisations · 6 postmortemsThe object store became the quorum: ten years of Elasticsearch, read from its own repository
A decade of one search engine's coordination design, reconstructed from two complete git histories, the design documents committed inside them, and a…
26 sources · 2 organisations · 3 postmortems