Clocks and Ordering

Definition

There is no global "now" in a distributed system. Machine clocks drift, are corrected by NTP in jumps that can move time backwards, and differ by milliseconds even when healthy. So an event with an earlier timestamp did not necessarily happen earlier.

The available mechanisms:

• Lamport clocks. A counter incremented on each event and carried on each message. Gives a total order consistent with causality, but cannot tell you whether two events were concurrent.
• Vector clocks. A counter per node. Can detect concurrency — which is exactly what you need to identify a conflict — at the cost of size proportional to the number of nodes.
• Consensus / a sequencer. Push all ordering decisions through one agreed sequence. Correct, and it costs a round trip.
• Tightly bounded physical clocks. Google's Spanner uses GPS and atomic clocks to bound clock uncertainty to a known interval, then deliberately waits out that interval before committing.

Why it matters

Almost every subtle distributed bug is an ordering bug. Last-write-wins on wall-clock timestamps silently discards writes when clocks disagree. A cache invalidation that arrives before the write it invalidates leaves the cache permanently stale. A "created_at" ordering in a UI shows events out of sequence, and no amount of database tuning fixes it.

Industry example

Spanner's TrueTime is the most instructive case because of what it chose to spend. Given an interval of clock uncertainty rather than a point in time, a transaction that wants an externally consistent timestamp waits until the uncertainty interval has passed before committing. The system deliberately adds latency — a few milliseconds — to purchase a global ordering guarantee.

The lesson is not "buy atomic clocks". It is that global ordering has a price, and the price is paid in latency on every write, whether you pay it with a commit wait, a consensus round trip, or a sequencer. Systems that appear to give global ordering for free are giving you something weaker and have not told you.

The corollary is the design move that matters far more often: scope ordering as narrowly as the business requires. Per-entity ordering — all events for one order, one user, one account — is cheap, achievable by partitioning on that key, and is what nearly every requirement actually means when it says "in order".

Failure scenarios

• Last-write-wins across regions with unsynchronised clocks: a write is silently lost because the other region's clock was 200 ms ahead.
• A negative duration in monitoring, because the end timestamp came from a different machine than the start.
• Ordering by database auto-increment ID, which reflects transaction start order rather than commit order, so a row can become visible after one with a higher ID.
• Assuming events arrive in the order sent, across any network, ever.

Trade-offs

Logical clocks are cheap and give you causality without physical time, but they cannot answer "what time was it". Physical time is human-meaningful and unreliable for ordering. Most robust designs carry both: a logical version for correctness decisions, and a wall-clock timestamp for humans, explicitly documented as advisory.

Interview question

"Two data centres both accept writes to the same record. Explain why last-write-wins by timestamp loses data, and what you would do instead."