Evidence ledger
23 sources
Checked 05 Sep 2026
Evidence ledger
One row per claim in Time cannot go backwards, and other lies production believed: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Tier grades follow the skill's eight-tier vocabulary. Checked dates are the day the source was read in this session (2026-09-05).
Access note for this session
This research session ran inside a remote environment with a network allowlist. Reachable and
fetched directly: github.com, raw.githubusercontent.com, pkg.go.dev, cloud.google.com,
plus PDFs pulled from GitHub-hosted mirrors. The following canonical hosts were blocked by the
egress policy: blog.cloudflare.com, azure.microsoft.com, engineering.fb.com,
developers.google.com, aws.amazon.com, martin.kleppmann.com, aphyr.com,
cse.buffalo.edu, web.archive.org. For sources on those hosts, content was read through
multiple independent search-index extractions in this session, cross-checked against each
other and against fetchable primary documents that quote them (for example, the Go monotonic
proposal quotes and links the Cloudflare postmortem directly). Every quote taken from a
blocked-host source below is marked [via extraction] and was confirmed by at least two
independent extractions. No quote below is reproduced from memory.
Consequence for the evidence mix: no conference-talk tier source could be fetched (USENIX and
YouTube are both blocked), so this guide carries zero talk cards; the talks that exist are
pointed to from the hunt section instead.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Cloudflare | How and why the leap second affected Cloudflare DNS | postmortem | 2017-01-01 | 2026-09-05 [via extraction] | https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ | The root cause was an assumption about time itself, not a coding slip | "the root cause of the bug that affected our DNS service is the belief that time cannot go backwards" |
| 2 | Cloudflare | (same) | postmortem | 2017-01-01 | 2026-09-05 [via extraction] | https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ | Blast radius and timeline | "at peak approximately 0.2% of DNS queries" affected, "less than 1% of all HTTP requests" errored; most affected machines patched in 90 minutes, worldwide fix by 06:45 UTC |
| 3 | Cloudflare | (same) | postmortem | 2017-01-01 | 2026-09-05 [via extraction] | https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ | Failure mechanism | upstream performance values computed from wall-clock subtraction went negative during the leap second; the weighted selection code fed the negative number to Go's rand.Int63n(), which panics on a negative argument |
| 4 | Microsoft | Summary of Windows Azure Service Disruption on Feb 29th, 2012 | postmortem | 2012-03-09 | 2026-09-05 [via extraction] | https://azure.microsoft.com/en-us/blog/summary-of-windows-azure-service-disruption-on-feb-29th-2012/ | The date bug and its mechanism | the guest agent "calculated the valid-to date by simply taking the current date and adding one to its year", producing February 29, 2013, an invalid date; certificate creation failed, VMs stalled in initialization |
| 5 | Microsoft | (same) | postmortem | 2012-03-09 | 2026-09-05 [via extraction] | https://azure.microsoft.com/en-us/blog/summary-of-windows-azure-service-disruption-on-feb-29th-2012/ | The cascade multiplier was the health machinery, not the bug | after three consecutive VM initialization failures the host was marked for "human investigate" (HI) state as suspected hardware failure; the bug was identified 2 hours 38 minutes after it first triggered |
| 6 | Linux kernel | commit 6b43ae8a: ntp: Fix leap-second hrtimer livelock | source | 2012 (merged) | 2026-09-05 fetched | https://github.com/torvalds/linux/commit/6b43ae8a619d17c4935c3320d2ef9e92bdeed05d | The kernel's own leap second insertion path could deadlock the machine | commit message describes the NTP-lock vs timekeeping-lock livelock between do_adjtimex() and the leap-second hrtimer; Reported-by/Tested-by: Sasha Levin, Diagnosed-by: Thomas Gleixner |
| 7 | Linux kernel | commit 4873fa07: timekeeping: Fix leapsecond triggered load spike issue | source | 2012-07 (merged) | 2026-09-05 fetched | https://github.com/torvalds/linux/commit/4873fa070ae84a4115f0b3c9dfabc224f1bc7c51 | After the June 2012 leap second, CLOCK_REALTIME timers fired early and applications spun | "timers based on CLOCK_REALTIME are either expiring a second early or late"; "The reported immediate workaround - $ date -s \"date\" - is causing a call to clock_was_set()" |
| 8 | Linux kernel | commit 5baefd6d: hrtimer: Update hrtimer base offsets each hrtimer_interrupt | source | 2012-07 (merged) | 2026-09-05 fetched | https://github.com/torvalds/linux/commit/5baefd6d84163443215f4a99f6a20f054ef11236 | Why the fix was subtle: a race window between setting time and updating offsets | "Any interrupt which arrives in that window will see the new time but operate on stale offsets"; "guarantees that timers are not expired early" |
| 9 | Go project | issue #12914: time: use monotonic clock to measure elapsed time | source | 2015-10-13 | 2026-09-05 fetched | https://github.com/golang/go/issues/12914 | The gap was known 15 months before the outage | "Go's standard library doesn't provide any API to access a monotonic clock source" (tsuna, opening comment); milestone Go1.9, Proposal-Accepted |
| 10 | Go project | Proposal: Monotonic Elapsed Time Measurements in Go (design/12914-monotonic.md) | adr | 2017-01-26 | 2026-09-05 fetched | https://github.com/golang/proposal/blob/master/design/12914-monotonic.md | The design problem, stated | "Comparison and subtraction of times observed by time.Now can return incorrect results if the system wall clock is reset between the two observations." |
| 11 | Go project | (same) | adr | 2017-01-26 | 2026-09-05 fetched | https://github.com/golang/proposal/blob/master/design/12914-monotonic.md | The outage is cited as the motivating incident | "A Go program measuring a negative elapsed time during a leap second caused CloudFlare's recent DNS outage." |
| 12 | Go project | (same) | adr | 2017-01-26 | 2026-09-05 fetched | https://github.com/golang/proposal/blob/master/design/12914-monotonic.md | Why the two-API design was rejected: correlated failure | "The most common clock reset in a well-run production setting is the leap second, which occurs simultaneously on all systems. When it does, all the copies of the program across the entire distributed system fail simultaneously, defeating any redundancy the system might have had." |
| 13 | Go project | (same) | adr | 2017-01-26 | 2026-09-05 fetched | https://github.com/golang/proposal/blob/master/design/12914-monotonic.md | Scale of silent fixes | "about 30% of the calls to time.Now (by source code appearance, not dynamic call count) are used for measuring elapsed time and should use the system monotonic clock"; corpus analysis: "71% unaffected", "29% fixed in event of wall clock time warps" |
| 14 | Go project | Go 1.9 release notes (doc/go1.9.html, release-branch.go1.9) | vendor | 2017 | 2026-09-05 fetched | https://github.com/golang/go/blob/release-branch.go1.9/doc/go1.9.html | The shipped behavior | "The time package now transparently tracks monotonic time in each Time value, making computing durations between two Time values a safe operation in the presence of wall clock adjustments." |
| 15 | Go project | package time documentation, Monotonic Clocks section | vendor | current | 2026-09-05 fetched | https://pkg.go.dev/time | The design rule in one sentence, and its boundary | "the wall clock is for telling time and the monotonic clock is for measuring time"; marshaling methods strip the monotonic reading because it has "no meaning outside the current process" |
| 16 | Rust project | issue #56612: Instant::now can go backward | source | 2018-12-07 | 2026-09-05 fetched | https://github.com/rust-lang/rust/issues/56612 | The same failure re-appears in another runtime, from hardware this time | opened by goffrie: Instant::now() returned decreasing values on Windows (QueryPerformanceCounter unreliable on some multi-core systems), panicking with "specified instant was later than self" |
| 17 | Rust project | PR #56988: std: Force Instant::now() to be monotonic | source | merged 2019-01-08 | 2026-09-05 fetched | https://github.com/rust-lang/rust/pull/56988 | Rust's first answer: paper over the platform in the runtime | alexcrichton: "We tried relying on OS/hardware/clock implementations, but those seem buggy enough that we can't rely on them in practice"; approach mirrors Firefox's |
| 18 | Rust project | PR #89926: Make Instant::{duration_since, elapsed, sub} saturating and remove workarounds | source | merged 2022-02-13 | 2026-09-05 fetched | https://github.com/rust-lang/rust/pull/89926 | The workaround itself became the cost; reversed after three years | the8472: "Currently we must choose between two poisons. One is high worst-case latency and jitter of Instant::now() due to explicit synchronization; see #83093 for benchmarks, the worst-case overhead is > 100x. The other is sporadic panics on specific, rare combinations of CPU/hypervisor/operating system due to platform bugs." |
| 19 | Kubernetes | client-go tools/leaderelection/leaderelection.go | source | current | 2026-09-05 fetched | https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go | Production leader election ships with the clock caveat written into the package comment | "This implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing)"; timestamps are captured locally, tolerating arbitrary clock skew but remaining sensitive to skew rate; tolerated rate ratio approximated by the LeaseDuration to RenewDeadline ratio |
| 20 | Martin Kleppmann | How to do distributed locking | blog | 2016-02-08 | 2026-09-05 [via extraction] | https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html | A lease plus a process pause violates mutual exclusion; the fix is a fencing token checked by the resource | a client can acquire a lease, pause (GC, page fault), and resume after expiry while another client holds the lock; a fencing token is a monotonically increasing number issued with the lock, and the storage service rejects writes carrying a token lower than one already seen; Redlock is criticized for depending on timing assumptions and lacking fencing tokens |
| 21 | CockroachDB | docs/design.md (time and HLC sections) | adr | maintained, read 2026 | 2026-09-05 fetched | https://github.com/cockroachdb/cockroach/blob/master/docs/design.md | How a database orders transactions on commodity clocks | "Each cockroach node maintains a hybrid logical clock (HLC)"; a transaction reads with an uncertainty interval up to "t+ε ... (ε is the maximum clock offset)"; conflicting data inside the interval "cause the transaction to abort and retry"; with better clocks it could "wait out the maximum clock offset after committing, but before returning to the client" as Spanner does |
| 22 | Cockroach Labs | Runbook template, clock-management.md | vendor | maintained, read 2026 | 2026-09-05 fetched | https://github.com/cockroachlabs/cockroachdb-runbook-template/blob/main/system-overview/clock-management.md | The operational contract: bounded offset or death | "The default maximum clock offset value is 500 ms."; a node detecting drift over 80% of max offset vs a majority of peers "spontaneously shuts down the node to guarantee database read consistency"; "Only use NTP sources that implement smearing or slewing of leap second" |
| 23 | Spanner: Google's Globally-Distributed Database (OSDI 2012) | paper | 2012-10 | 2026-09-05 fetched (PDF via Papers We Love mirror) | https://github.com/papers-we-love/papers-we-love/blob/main/datastores/spanner-google%27s-globally-distributed-database.pdf | The measured cost of bounded uncertainty | "ε is typically a sawtooth function of time, varying from about 1 to 7 ms over each poll interval. ε is therefore 4 ms most of the time."; applied drift rate 200 microseconds/second; poll interval 30 s; commit wait measured at about 5 ms; "bad CPUs are 6 times more likely than bad clocks" | |
| 24 | Making every (leap) second count with our new public NTP servers | blog | 2016-11-30 | 2026-09-05 fetched | https://cloud.google.com/blog/products/gcp/making-every-leap-second-count-with-our-new-public-ntp-servers | The smear, quantified | "we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and 'smear' the extra second across these twenty hours"; "For timekeeping purposes, December 31 will seem like any other day." (Michael Shields, Technical Lead, Time Team) | |
| 25 | Leap Smear (Public NTP documentation) | vendor | current | 2026-09-05 [via extraction] | https://developers.google.com/time/smear | The proposed standard smear, and its adoption | recommended standard is a 24-hour linear smear from noon to noon UTC; the frequency change is about 11.6 ppm, within the thermal error of most quartz oscillators; Amazon uses this smear in AWS; Google previously used 20 hours and changed to align | |
| 26 | Meta | It's time to leave the leap second in the past | blog | 2022-07-25 | 2026-09-05 [via extraction] | https://engineering.fb.com/2022/07/25/production-engineering/its-time-to-leave-the-leap-second-in-the-past/ | A second large operator smears differently, and wants the mechanism abolished | Meta smears the leap second over 17 hours starting at 00:00:00 UTC based on tzdata; UTC has been adjusted 27 times since leap seconds were introduced; "Introducing new leap seconds is a risky practice that does more harm than good" (Obleukhov and Byagowi) |
| 27 | AWS | ClockBound (README) | source | maintained, read 2026 | 2026-09-05 fetched | https://github.com/aws/clock-bound | Bounded uncertainty as a commodity API | "The window of uncertainty (the Clock Error Bound) is defined by two timestamps (earliest, latest) within which true time exists"; "the clock error bound grows in between clock updates"; designed to pair with the Amazon Time Sync Service |
| 28 | AWS | Amazon Time Sync Service microsecond-accurate time announcement | vendor | 2023-11 | 2026-09-05 [via extraction] | https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-time-sync-service-microsecond-accurate-time/ | Vendor claim: microsecond-level sync on supported EC2 instances | the service "synchronizes time within microseconds of UTC" using the Nitro System and GPS-disciplined reference clocks; clock accuracy claimed in the low double-digit microsecond range within the guest OS (vendor claim, not independently measured here) |
| 29 | Kulkarni, Demirbas, Madeppa, Avva, Leone | Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases | paper | 2014-05 | 2026-09-05 [via extraction; mechanism corroborated by fetched CockroachDB design.md, which cites and implements it] | https://cse.buffalo.edu/tech-reports/2014-04.pdf | The HLC construction CockroachDB builds on | HLC combines the causality tracking of logical clocks with proximity to physical NTP time, so one timestamp can both order causally-related events and identify consistent snapshots |
| 30 | Kyle Kingsbury (aphyr) | The trouble with timestamps | blog | 2013-10 | 2026-09-05 [via extraction] | https://aphyr.com/posts/299-the-trouble-with-timestamps | Last-write-wins on wall-clock timestamps discards concurrent writes; skewed clocks make the discard arbitrary | Jepsen's analyses of LWW stores demonstrate acknowledged-write loss under concurrency even with synchronized clocks; with skewed clocks, the "winning" write is chosen by whichever node's clock reads later (figures vary by test and are not re-quoted here because the primary page could not be fetched in this session) |
What could not be verified in this session
- The full comment thread of golang/go#12914 (the GitHub issue UI returned only the opening post to this session's fetcher). The opening date, milestone, and acceptance labels were fetched; the proposal document carries the design argument.
- Exact merge dates for the three Linux kernel commits (the commit pages rendered without visible dates to this session's fetcher). The July 2012 dating of commits 4873fa07 and 5baefd6d is corroborated by the patch postings on kernel mailing lists surfaced in search and by contemporaneous press coverage of the 2012-06-30 leap second.
- Talk-tier sources (USENIX, YouTube, Strange Loop hosts all blocked). Named in the hunt section rather than cited.