An event-processing platform orders records by the timestamp the producing service attached. Records occasionally appear out of order and some appear to arrive before they were created. What is happening and how should ordering be established?

What is happening

Wall clocks on different machines disagree, typically by milliseconds and occasionally by much more when synchronisation fails, a virtual machine is migrated, or a clock steps backwards. Ordering events from multiple producers by their local wall clock is ordering by an unreliable measurement, and at high event rates the disagreement is larger than the intervals you are trying to order.

Events that appear to arrive before they were created are the same phenomenon seen from the consumer side: the consumer's clock is ahead of the producer's, or the producer's stepped backwards.

The distinctions that resolve it

• Event time versus processing time. Event time is when it happened according to the producer; processing time is when your system saw it. They are different, both are useful, and conflating them causes most time-related bugs in stream processing. Store both.
• Causal ordering versus temporal ordering. Usually you do not need to know which of two events happened first in absolute time — you need to know whether one caused the other. Causality can be captured exactly, with sequence numbers per producer or a logical clock, and does not depend on synchronisation at all.
• Per-key ordering versus global ordering. The invariant is almost always per entity: this account's transactions must be ordered. Establishing that with a per-key sequence number from a single writer is cheap and exact. Global ordering is expensive and rarely required.

What to do

• Assign a monotonic sequence per key at the single writer for that key. This is the authoritative ordering, it is exact, and it survives clock problems entirely.
• Keep wall-clock timestamps for human interpretation and analytics, and treat them as approximate. Never use them for correctness decisions.
• Where events genuinely come from multiple writers, use a logical clock or accept a bounded-staleness model with an explicit lateness window and a defined behaviour for late arrivals — dropped, or routed to a correction path.
• Monitor clock skew as an operational metric. Most teams discover their clocks are wrong through a data bug rather than through an alert, which means the bug is old by the time it is found.

The failure this prevents

The dangerous version is silent: a balance computed by applying transactions in timestamp order produces the wrong result when two arrive out of order, and the arithmetic still works. Nothing errors, the number is simply wrong, and it stays wrong until a reconciliation notices. That is the argument for making ordering an explicit property of the data rather than something inferred from a measurement.