Every machine's opinion of the current time is wrong by an unknown, changing amount that occasionally jumps. This guide reconstructs how production systems live with that fact, from two postmortems, three Linux kernel commits, the Go and Rust standard library design records, Kubernetes leader election, and the clock architectures published by Google, Meta, AWS and CockroachDB. Afterwards you can audit a codebase for the wall-clock subtraction bug, decide when a lease needs a fencing token, and price bounded uncertainty against uncertainty restarts for your own ordering problem.
The problem, stated without naming a clock API: two machines must agree which of two events came first, and one machine must know how long something took, while the instrument both questions depend on is being silently adjusted underneath them.
A computer carries two instruments that both call themselves "the time". The wall clock
(CLOCK_REALTIME) answers "what time is it", is disciplined by an outside
authority through a sync daemon, and therefore jumps: forwards, backwards, or sideways at
11.6 parts per million during a leap smear. The monotonic clock
(CLOCK_MONOTONIC) answers "how long has it been", counts from boot, promises
only to never decrease, and means nothing outside the process that read it. Go's time
package documentation compresses the whole field into one sentence: the wall clock is for
telling time, the monotonic clock is for measuring time
[15].
Nearly every incident in this guide is a system reaching for the wrong instrument, or assuming one instrument has the other's properties. Cloudflare subtracted two wall-clock readings and got a negative duration [1]. Azure did calendar arithmetic by incrementing a year field and manufactured a date that does not exist [2]. The Linux kernel itself, the thing that implements both clocks, deadlocked and then spin-stormed on the leap seconds of 2012 [3, 4]. And every last-write-wins datastore quietly turns "which node's clock reads later" into "whose data survives" [22].
The industry's chosen fix for the leap second is to make every clock deliberately wrong, and no two operators make them wrong the same way. Google smears the extra second linearly over 24 hours, noon to noon [18]; in 2016 it used 20 hours [17]; Meta smears over 17 hours starting at midnight [19]. During a leap event, two fleets, each perfectly synchronised to its own provider, can disagree by most of a second. That is why CockroachDB's operations guidance flatly requires all nodes to use smeared time sources from one regime [16]: the danger is not the leap second, it is mixing answers about it.
Scope. This guide covers the failure modes of trusting machine clocks and the four production answers to them: the monotonic split, smearing, leases with fencing, and bounded uncertainty. It deliberately does not cover NTP or PTP protocol engineering and security, GPS spoofing, timezone and tzdata handling, or the regulatory clock-sync mandates in finance. Those are real topics with their own records; they are not this dig.
The common shape across Google, AWS, Meta, CockroachDB and the two language runtimes: a distribution layer that makes clocks less wrong, a runtime layer that stops you holding the wrong clock, and a coordination layer that decides how much wrongness the design absorbs.
Distribution. Google's TrueTime feeds every datacenter from GPS receivers plus a minority of atomic-clock "Armageddon masters", chosen because the two reference types fail in uncorrelated ways; daemons poll a mix of nearby and far masters every 30 seconds and advertise a worst-case error, assuming local drift of at most 200 microseconds per second between polls [14]. AWS ships the same idea as a managed commodity: the Time Sync Service claims synchronisation "within microseconds of UTC" on supported EC2 instances (a vendor claim; no independent measurement appears in this corpus) [21], and the open-source ClockBound daemon exposes it as a pair of timestamps, earliest and latest, "within which true time exists", a bound that visibly grows between clock updates [20]. CockroachDB's operations guidance shows the floor of this layer: chrony against smeared sources, a configured maximum offset of 500 ms, and a node that finds itself more than 80% of that away from a majority of its peers shuts down rather than serve inconsistent reads [16].
Runtime. This layer exists because application programmers will
otherwise subtract wall-clock readings; the Go corpus analysis put that mistake in roughly
30% of time.Now call sites [10]. Go's answer, shipped
in Go 1.9 after the design record discussed in section 3, is to hide both readings inside
one value: time.Now captures wall and monotonic together, time-telling
operations use the wall reading, and subtraction or comparison silently use the monotonic
one [13, 15]. Rust keeps two visible
types, SystemTime and Instant, and spent 2018 to 2022 learning
what to do when the operating system's "monotonic" clock is not
[6, 7, 8].
Coordination. Three positions, in increasing order of engineering and latency cost. Order by raw timestamp and accept that concurrent or skewed writes are silently discarded; this is last-write-wins, and Kyle Kingsbury's Jepsen work demonstrated acknowledged-write loss under it years ago [22]. Grant time-limited leadership with leases, and either accept the fencing gap or make the protected resource check a monotonically increasing token, the repair Martin Kleppmann argued for in 2016 [12]; Kubernetes ships the un-fenced version with the caveat written into the package comment [11]. Or carry the clock error as data: Spanner assigns a commit timestamp and then waits out the uncertainty (about 5 ms measured in the paper's benchmarks) so the timestamp is guaranteed past before anyone sees it, while CockroachDB, on 500 ms commodity bounds where waiting is unaffordable, instead restarts any transaction that reads a value inside its uncertainty interval [14, 13].
One clock for meaning, one for arithmetic. The runtime either hides the split (Go) or makes it a type distinction (Rust). Either way, a duration must never be computed from two wall readings.
Recorded at: Go proposal, rust-lang #56612
Leap seconds are absorbed by lying smoothly: run every clock slightly slow for hours instead of stepping once. The lie is bounded, published, and different per provider, which makes provider mixing the new hazard.
Instead of one timestamp, an interval [earliest, latest] the truth is inside. Systems then buy correctness by waiting the interval out, or by retrying anything that lands in it.
Recorded at: ClockBound, CockroachDB design.md
Four forks in the road, each with a recorded argument behind it, and the condition that flips the answer.
Instant subtraction saturate to zero and delete the enforcement machinery [8]| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Go time API | Two readings in one value | Two visible APIs | Misuse fails everywhere at once, on a leap second | Go proposal, 2017 |
| Rust Instant on buggy platforms | Saturate to zero | Global enforcement, panics | Enforcement cost >100x worst case; panics tracked platform bugs | PR #89926, 2022 |
| Leap second handling | Smear over hours | Step at midnight | The step path broke the kernel twice in 2012 | kernel commit 4873fa07 |
| Cross-node ordering | ε-aware timestamps or fenced leases | Raw last write wins | LWW converts clock skew into silent data loss | Kingsbury, 2013 |
| Wait or retry on ε | Spanner waits; CockroachDB retries | The other one, respectively | Purchasable ε of 4 ms vs commodity 500 ms | design.md |
Three failure classes account for the public record: the backwards subtraction, the impossible date, and the kernel's own time step. The fourth class, silent last-write-wins loss, has no postmortem, and that absence is a finding.
rand.Int63n, which panics on negative input.date -s "`date`".The most common clock reset in a well-run production setting is the leap second, which occurs simultaneously on all systems. When it does, all the copies of the program across the entire distributed system fail simultaneously, defeating any redundancy the system might have had. Russ Cox, Go monotonic time proposal, January 2017
Everything quantitative in the corpus, dated and attributed. Measured unless marked as a vendor claim.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Clock uncertainty ε | 1–7 ms, avg ~4 ms | Sawtooth per 30 s poll, GPS + atomic references; measured | 2012 | Spanner paper | |
| Assumed worst-case local drift | 200 µs/s | The bound ε is derived from; "bad CPUs are 6 times more likely than bad clocks" | 2012 | Spanner paper | |
| Commit wait | ~5 ms | Measured, 1-replica microbenchmark; the price of external consistency | 2012 | Spanner paper | |
| Default max clock offset | 500 ms | CockroachDB | Commodity NTP assumption; consistency depends on staying inside it | 2026 | runbook |
| Self-termination threshold | 80% of max offset | CockroachDB | Node shuts down when drifted vs a majority of peers | 2026 | runbook |
| Leap smear, 2016 event | 0.0014% for 20 h | Clocks run slow either side of the leap second | 2016 | Google blog | |
| Standard smear | 24 h, ~11.6 ppm | Google, AWS | Linear, noon to noon; within quartz thermal error | current | smear doc |
| Meta smear | 17 h from 00:00 UTC | Meta | tzdata-driven; a third, different regime | 2022 | Meta blog |
| Cost of enforced monotonicity | >100× worst case | Rust std | Synchronised Instant::now overhead and jitter; why it was removed | 2021 | PR #89926 |
| Wrong-clock call sites | ~30% of time.Now | Go corpus | By source appearance; 29% of uses auto-fixed by the proposal | 2017 | Go proposal |
| Cloudflare impact | 0.2% DNS, <1% HTTP | Cloudflare | Peak error rates; 90 min to patch worst machines | 2017 | postmortem |
| Azure detection gap | 2 h 38 min | Microsoft | First trigger to bug identification, leap day 2012 | 2012 | postmortem |
| Cloud clock sync | "within microseconds" | AWS | Vendor claim for Time Sync on supported instances; not independently measured here | 2023 | AWS announcement |
The Spanner figures are 2012 measurements of purpose-built hardware; treat them as a floor for what money can buy, not what your fleet has. The AWS microsecond figure is a vendor claim with no independent measurement in this corpus. The CockroachDB 500 ms is a default, not a measurement of your NTP; the runbook's whole point is that you must measure. And the 30% Go figure counts call sites, not call volume.
Every source behind this page, graded. This corpus is unusually strong in source-code records (the arguments happened in public issue trackers) and carries no talk tier: the relevant talks exist but sit on hosts this research session could not reach, so they are pointed to in section 8 rather than cited as read.
The canonical backwards-time incident: a negative wall-clock subtraction stored as state, a panic three steps later, and a fleet failing simultaneously. Names its own root cause as a belief, not a bug.
Year+1 date arithmetic met February 29. The lasting lesson is the amplifier: health automation read a software bug as mass hardware failure and quarantined healthy hosts.
The leap-second insertion path could deadlock against the timekeeping locks. The subsystem that implements clocks is not immune to clock bugs.
After the June 2012 leap second, CLOCK_REALTIME hrtimers fired a second early and
applications spun. The commit message preserves the industry-wide workaround,
date -s "`date`", as a fossil of the night.
The request that sat for 15 months: Go offered no monotonic source, so measuring an operation reliably required platform-specific code. Milestoned to Go 1.9 only after the 2017 outage.
Russ Cox's design record: cites the Cloudflare outage as motivation, quantifies the corpus (30% of call sites measure elapsed time), and rejects the two-API design because its failures are simultaneous fleet-wide, not random.
The shipped outcome: "the time package now transparently tracks monotonic time in each Time value, making computing durations between two Time values a safe operation in the presence of wall clock adjustments."
The operating rule and its boundary: subtraction and comparison use the monotonic reading; serialisation strips it because it has no meaning outside the process.
The monotonic clock itself lying: QueryPerformanceCounter regressing on some Windows multi-core systems, panicking Rust programs with "specified instant was later than self".
The first answer: a global watermark in std, mirroring Firefox, because platform clocks "seem buggy enough that we can't rely on them in practice".
The reversal, three years later: "we must choose between two poisons", enforcement costing over 100x worst-case, versus rare panics from platform bugs. Rust chose the quiet poison and saturates to zero.
Production leader election that says the quiet part in its doc comment: "this implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing)". Tolerates arbitrary skew, is sensitive to skew rate.
The GC-pause argument: a client can acquire a lease, stall past expiry, and resume convinced it still holds the lock. The repair is a fencing token, a monotonically increasing number the protected resource itself checks. Also the source of the efficiency-versus-correctness framing in the Redlock debate.
The public design record of ordering on commodity clocks: per-node HLCs, transaction reads carrying an interval up to t+ε, conflicts inside it forcing a retry, and an explicit note that with better clocks it would commit-wait like Spanner instead.
The operational contract behind the design: 500 ms default max offset, node suicide at 80% drift versus a majority, chrony recommended, and only smeared or slewed leap second sources permitted across a cluster.
TrueTime measured in production: ε a sawtooth of 1 to 7 ms, 4 ms typical, derived from an assumed 200 µs/s worst-case drift; commit wait around 5 ms; GPS and atomic references chosen for uncorrelated failure. Read via the Papers We Love mirror of the OSDI publication.
The hybrid logical clock: causality tracking that stays close to physical NTP time, so one timestamp orders related events and cuts consistent snapshots. The construction CockroachDB's design doc cites and implements. (Paper host unreachable in this session; mechanism corroborated through the fetched design doc.)
The smear, first-party: clocks 0.0014% slower for ten hours either side of the leap second, so "December 31 will seem like any other day".
The proposed standard: 24-hour linear smear, noon to noon, roughly 11.6 ppm, adopted by AWS; Google itself moved from 20 hours to align. Standardisation is explicitly the goal because divergent smears disagree.
Meta's position paper: 27 leap seconds so far, each one an industry incident; Meta smears over 17 hours from midnight UTC; "introducing new leap seconds is a risky practice that does more harm than good".
Bounded uncertainty as an open-source commodity: every reading is (earliest, latest) "within which true time exists", and the bound honestly grows between clock updates.
The claim that changes the economics: Nitro-based, GPS-disciplined clocks synchronised "within microseconds of UTC" on supported instances, at no extra charge. Unverified independently in this corpus.
Why last-write-wins plus wall clocks is a data-loss design: timestamps decide conflicts, clocks decide timestamps, and skew decides clocks. Jepsen's controlled tests made the loss visible; production dashboards do not.
Six rungs from demonstrating the lie to operating under it. The crossing from toy to real is rung four.
Write a 30-line service that timestamps request pairs with the wall clock and computes
latency. Run it under libfaketime or step the clock in a VM with
date -s, and watch negative durations appear. Port it to a monotonic source
and repeat.
Done when: you have logged a negative duration from the wall-clock version and cannot from the monotonic one. Teaches: the two-instruments split as an experience, not a rule.
Pick a service you own. Find every duration computed from wall-clock reads: in
pre-1.9-style Go, System.currentTimeMillis() deltas on the JVM,
new Date() arithmetic in Node, time.time() subtraction in
Python. Classify each call site as telling or measuring, the way Cox's corpus analysis
did.
Done when: you have a count and a fix list, and know your codebase's version of the 30% number. Teaches: how invisible this class is in review.
Implement a lock with a TTL in any store. Two clients contend; pause the holder (SIGSTOP, or a debugger breakpoint) past expiry, then resume it and let it write. Watch both clients act as leader, exactly as the Kubernetes package comment warns.
Done when: you have two "leaders" writing interleaved output on demand. Teaches: a lease is a clock bet, and the process cannot referee its own pause.
Issue a monotonically increasing token with each lock grant and make the protected resource reject writes bearing a token at or below the highest seen. Re-run rung three's pause attack.
Done when: the stale leader's write is refused by the resource, not by the leader's own good behaviour. Teaches: exclusivity is enforced at the resource or not at all. This is the toy-to-real crossing.
Two nodes with deliberately skewed clocks exchange messages, each carrying an HLC timestamp per the Kulkarni construction. Verify that message order is respected in HLC order despite the skew, and that HLC time stays near wall time.
Done when: a causally-later write always carries a larger HLC timestamp under 10 s of injected skew. Teaches: causality is orderable without trusting either clock.
On EC2, install ClockBound and log (earliest, latest) across instances; elsewhere, simulate a bound from NTP offset stats. Implement one commit-wait: hold a write invisible until latest-at-write has passed everywhere. Measure the added latency and compare it to your ε.
Done when: injected skew inside the bound never produces an out-of-order read, and you can state the latency bill. Teaches: the Spanner trade as a number on your own hardware.
The queries that found this material, copyable. The last block names the talks this session could verify exist but not watch; they are the natural next layer.
cloudflare leap second postmortem "time went backwards"azure "leap day" 2012 "service disruption" guest agent certificateleap second outage 2012 linux futex reddit mozilla"clock skew" postmortem "we" -tutorialgolang issue 12914 monotonic clock proposalrust "Instant::now can go backward" saturatingtorvalds commit "leap-second hrtimer livelock"leaderelection.go "a.k.a. fencing" kubernetes"leap smear" site:developers.google.com OR site:cloud.google.comcockroachdb runbook "clock management" max offset chronyspanner truetime epsilon "commit wait" osdi pdfaws clockbound github "earliest" "latest" bound"hybrid logical clock" kulkarni demirbas consistent snapshots"keeping time in real systems" kavya joshi strange loop 2017"how to do distributed locking" kleppmann fencing token redlock antirez replysreconf OR usenix "precision time protocol" meta oleg obleukhov talkjepsen cassandra "last write wins" acknowledged writes lostThis page was researched from a network environment
with a restricted egress allowlist. Sources marked † sit on hosts that allowlist
blocked; their content was read through multiple independent search-index extractions,
cross-checked against each other and against the fetchable documents that quote them, and
every quotation from them in this page appears in at least two of those extractions. The
full per-claim ledger, including access notes, ships beside this file as
sources.md.