Logging

Definition

Logs are timestamped records of discrete events. Their value is in answering questions you did not anticipate, which is exactly what metrics cannot do — and their cost grows linearly with traffic, which is exactly what metrics avoid.

What separates useful logging from noise

Structured, not prose. {"event":"payment_failed","order_id":"x","provider":"y","code":"z"} can be filtered, grouped and counted. "Payment failed for order x" can be grepped, badly. Structure is the difference between an investigation taking two minutes and two hours.

A correlation ID on every line, injected at the edge and propagated through every hop, including asynchronous ones. Without it, logs from nine services are nine unrelated streams.

Consistent field names across services. user_id in one and userId in another means no cross-service query works. This is a platform decision, not a per-team one.

Levels that mean something. ERROR should mean "a human should look at this". If normal operation produces errors, the level is wrong and the signal is dead.

Context at the point of failure. The inputs, the identifiers, the decision taken. A stack trace without the parameters that produced it costs an hour of reconstruction.

What not to log

• Personal data, credentials, tokens, card numbers. Logs are widely readable, long-lived and frequently exported. This is the most common source of accidental data exposure, and it is usually discovered in an audit rather than a breach.
• Full request and response bodies by default. Enormous, and the fastest route to logging something sensitive by accident.
• Successful health checks. They will dominate the volume and inform nothing.

The cost problem

At scale, log volume becomes a material budget line, and it is one of the few costs that grows exactly with success. The responses, in order of value:

1. Log less at the source. One structured event per request beats twelve prose lines.
2. Sample the successful path, keep everything for errors. A 1% sample of successes plus 100% of failures preserves nearly all the diagnostic value at a fraction of the cost.
3. Tier retention — days in hot searchable storage, months in cheap object storage.
4. Aggregate what is really a metric. Counting occurrences by logging every one is the expensive way to build a counter.

Failure scenarios

• Logging on the request's critical path, synchronously, to a remote system — so a logging outage becomes an application outage.
• Unstructured logs that cannot be queried when it matters.
• Correlation ID lost at an asynchronous boundary, so the trace stops at the queue.
• Debug logging enabled in production "temporarily", generating a bill and a performance problem.
• No log at the decision point. The system chose a branch and there is no record of why.

Interview question

"Your logging bill has tripled with traffic. Name four things you would change, in order."