beginner 2 min answer

Two services in different regions log the same failed payment, and the merged log view shows the settlement error 50 minutes before the request that caused it. Both lines have timestamps. What is wrong with them, and what should every log line carry instead?

loggingtimestampsutcclock skewordering
Show the full answer Hide the answer

What is being tested

Whether the candidate knows that a timestamp is only useful if every reader can place it on one timeline, and that a log line whose time cannot be compared is a line you have to guess about.

The mechanism

The usual cause is a timestamp without a zone. One service writes 2026-03-08 14:02:11 in the host's local time (say UTC-5), the other writes the same shape in UTC. The log store parses both as if they were in one zone, and a five-hour offset appears, or a 50-minute one when a half-hour zone is involved or a daylight-saving shift happened on one host only. Free-text formats like 03/04/26 add a second ambiguity, because the day and month order depends on the library's locale.

The second cause is subtler. Even with correct UTC on both, hosts disagree by milliseconds to seconds when NTP is unhealthy, so sorting by timestamp can still put an effect before its cause by a small margin.

What every line should carry

  • An absolute time in UTC with the zone written out, in the ISO 8601 form 2026-03-08T14:02:11.123Z. Millisecond precision at least. It sorts correctly as plain text, and it has no daylight-saving hour that happens twice.
  • A request or correlation identifier, because ordering within one request should come from causality (parent and child), not from comparing clocks on different machines.
  • The emitting host or pod name, so skew can be diagnosed: if one host is consistently 40 s off, every line from it is suspect.

The decision rule

Log in UTC always, and convert to local time only in the viewer. Translating at the edge of the display is free; translating at ingest is a bug waiting for a daylight-saving weekend.

When this is the wrong answer

Human-facing audit trails sometimes need the local wall-clock time of an event as the user saw it (a shift handover at a depot). Store that as a separate field alongside the UTC instant, never instead of it.

Common weak answers

  • "Sort by timestamp, then it is fine." Sorting only works if the clocks agree, and across hosts they never agree exactly.
  • "Use epoch seconds." Unambiguous but unreadable during an incident, and second resolution collapses a burst of 400 events into one tick.
  • "Fix it in the dashboard by adding five hours." It works until a service moves region or the clocks change, and then it silently lies.

What a strong answer adds

A strong answer says the check is cheap and automatable: a lint or a log-pipeline test that rejects any line with a timestamp lacking Z or an offset, and an alert on host clock offset above, say, 100 ms. It also notes that the same discipline applies to metrics and traces, because a trace waterfall built from skewed clocks shows a child starting before its parent and sends the investigation the wrong way.