Two regions both accept writes to the same user record and resolve conflicts by last-write-wins on timestamp. Users report edits silently disappearing. Explain and fix.

What is being tested

Whether you know that wall-clock timestamps cannot order events across machines, and whether you can name a conflict-resolution strategy that does not lose data.

The mechanism

There is no global "now". Machine clocks drift, and NTP corrects them in jumps that can move time backwards. Two regions' clocks can differ by tens or hundreds of milliseconds while both appear healthy.

So when region A writes at its local 10:00:00.150 and region B writes at its local 10:00:00.100, the second write in real time may carry the earlier timestamp. Last-write-wins then discards the newer edit. Nothing errors. The user's change simply is not there.

Worse, the loss is silent and unattributable: by the time the user notices, there is no record that a conflict occurred at all. This is the property that makes clock-based LWW so dangerous — it does not fail loudly, it fails quietly and permanently.

The fixes, in increasing order of fidelity

1. Version vectors. Each replica keeps a counter. A write carries the version it observed. On receipt, you can distinguish three cases: the incoming write descends from the local one (accept it), the local one descends from it (ignore), or they are concurrent (a genuine conflict). The crucial gain over timestamps is that concurrency becomes detectable rather than silently resolved.

2. Keep both versions and resolve deliberately. Once a conflict is detected, you have choices LWW never gave you: merge them if the data type allows, keep both and let the user choose, or apply a domain rule. Dropbox-style file synchronisation takes the honest route for a case where automatic merging is impossible — it keeps both and names one as a conflicted copy. Users find that mildly annoying; they would find silent data loss unacceptable.

3. Conflict-free replicated data types. For structures with a mathematical merge — counters, sets, some text — CRDTs converge automatically with no coordination. Powerful where they fit, and they do not fit arbitrary business objects.

4. Single-writer per record. Route all writes for a given user to one home region. Conflicts cannot occur because concurrent writes cannot occur. This is by far the simplest correct answer and is under-used because it feels like a retreat from active-active. It is not: it is active-active at the system level with a deterministic owner per record.

5. Consensus. Serialise writes through a quorum. Correct, and it costs a cross-region round trip on every write, which is usually the thing active-active was meant to avoid.

The general lesson

Global ordering has a price, and it is always paid in latency — a commit wait, a consensus round trip, or a sequencer. Google's Spanner makes this unusually explicit: given a bounded clock uncertainty interval, it deliberately waits out the interval before committing, buying external consistency with a few milliseconds on every write. Any system that appears to give global ordering for free is giving you something weaker and has not said so.

The practical design move, far more often applicable than any of the above: scope ordering as narrowly as the business requires. Per-entity ordering is cheap, achievable by routing or partitioning on that key, and is what almost every "must be in order" requirement actually means.

Related traps worth naming

• Ordering by auto-increment ID, which reflects transaction start order, so a row can become visible after one with a higher ID.
• Negative durations in monitoring, because start and end timestamps came from different machines.
• Assuming messages arrive in the order sent, across any network, ever.