Your logging bill has tripled alongside traffic growth. Name four changes in order of effectiveness.
Show the full answer Hide the answer
What is being tested
Whether you attack the source before the storage, and whether you understand that log cost is one of the few costs that scales exactly with success.
The four changes, in order
1. Log less at the source. One structured event per request replaces twelve prose lines. Delete successful health checks, which will otherwise dominate the volume and inform nothing. Delete the debug logging somebody enabled "temporarily" during an incident four months ago. This is the largest and cheapest reduction, and it improves signal at the same time.
2. Sample the successful path, keep 100% of errors. A 1% sample of successful requests plus every failure preserves nearly all diagnostic value at a fraction of the cost. The refinement worth adding: sample by trace, not by line, so a sampled request keeps its complete story rather than a random scattering of lines.
3. Tier retention. Days in hot searchable storage, months in cheap object storage queried more slowly. Most logs are never read; indexing everything is paying for search on data nobody searches. Retention should differ by class — audit logs may need years, debug logs need days — which requires classifying at emission rather than at storage where everything looks the same.
4. Move counters into metrics. Counting occurrences by logging every one is the expensive way to build a counter. If the only thing anyone does with a log line is count how many there were, it should be a metric — cost independent of volume rather than linear in it.
What not to do
Reduce retention uniformly to a few days. It saves money and destroys the ability to investigate anything discovered late, which is most things. Tiering achieves the saving without the loss.
The architectural points worth raising
Never log synchronously to a remote system on the request path. A logging outage then becomes an application outage, and a slow log sink adds latency to every request. Write to standard output or a local file; a local agent buffers and ships.
Check what is being logged. A volume investigation frequently discovers personal data, credentials or full request bodies in the logs — which is a security finding, not a cost one, and it is usually the more urgent of the two.
What a strong answer adds
Noting that the cost is a symptom of missing structure. Teams log verbosely because unstructured logs require volume to be useful — you log everything because you cannot query precisely. Structured logging with consistent field names and a correlation ID on every line makes a fraction of the volume more useful than the whole of it was.