A service emits one INFO log line per request and increments one counter per request. At 3,000 requests per second the two describe exactly the same traffic. Why does keeping the log cost orders of magnitude more than keeping the counter, and when is the log still the right thing to emit?
Show the full answer Hide the answer
The mechanism
A metric is aggregated at write time; a log is not. The counter lives in the process as a single number. Whatever happens between one collection interval and the next — one request or a million — the wire carries one sample per series per interval. The aggregation has already happened, in memory, for free.
A log line is one record per event. Nothing is collapsed, because collapsing is exactly what you refused to do when you chose a log: the point of the line is that it retains the particulars of one request, and particulars cannot be summed.
The arithmetic
At 3,000 rps with a 15-second collection interval, the counter produces 4 samples a minute, about 5,800 a day, for one series. Time-series stores compress successive samples of a slow-moving counter to a small number of bytes each, because timestamps are regular and values are near-monotonic. A single series costs tens of kilobytes a day.
The same traffic as logs is 3,000 × 86,400 ≈ 260 million lines a day. At 400 bytes a line that is roughly 100 GB a day before indexing, and the index that makes the lines searchable commonly adds a further 30% to 100% on top. Keeping it for 30 days is several terabytes of hot, indexed storage.
The ratio is not about bytes on the wire. It is about how many records exist, and that is decided by whether aggregation happened before storage or after.
When the log is still right
- When the question is "which one". A counter tells you 1.2% of checkouts failed. It structurally cannot tell you which tenant, which build or which card issuer, because those dimensions were discarded before storage. Adding them as labels multiplies the series count by their cardinality, which is why metrics systems forbid high-cardinality labels.
- When the event is rare and consequential. A payment authorisation, a permission change, a schema migration: hundreds a day, not hundreds of millions. Volume is the whole cost driver, so rare events are nearly free to log in full.
- When you will need evidence, not a number. Audit and dispute resolution need the record, and a counter is not a record.
The decision rule: emit a metric for every question you already know you will ask, and a log line for the events whose details you will need to reconstruct one at a time. For the middle ground — high volume where you still need particulars — the answer is neither: sample, or emit one wide structured event per request and aggregate it at query time.
Common weak answers
- "Logs are text and text is big." Format is a rounding error. A 400-byte line compresses to under 100 bytes; the cost is that there are 260 million of them and that they are indexed.
- "Just turn the log level up to WARN." That works, and it is worth doing, but it answers a different question. The reason INFO-per-request is expensive is not that it is chatty; it is that it is per-request.
- "Use metrics for everything." Then the first incident that affects one customer on one build is undebuggable, because the dimension that identifies it was never stored.