A discussion platform's log volume grows faster than its traffic and now costs more than the compute generating it. What is driving the growth, and what should change?

What drives the growth

Log volume grows superlinearly with traffic for reasons that compound:

1. Log lines added and never removed. Every incident adds debug logging that stays forever. Nobody deletes a log line, because it might be needed.

2. Verbosity in libraries and frameworks, often enabled by default and never reviewed.

3. Log lines inside loops, which scale with data size rather than with request count — one request processing a thousand items produces a thousand lines.

4. Service count. Splitting a monolith into forty services means each request produces logs at forty places, plus the infrastructure logs of forty deployments.

5. Structured logging adopted without discipline, where every line carries a large context object, so each line is many times bigger.

What should change

1. Classify logs by purpose, and route each purpose differently. - Audit — must be retained, cannot be sampled, goes to durable cheap storage. - Debug — high volume, low value after a few days, aggressively sampled and briefly retained. - Error — never sampled, retained longer. - Request logs — replaced by traces and metrics wherever possible.

2. Sample debug and info logs, with a rule that anything belonging to an errored or slow request is kept in full. Head-based sampling of ordinary traffic plus tail-based retention of interesting traffic gives most of the value at a fraction of the volume.

3. Move counting out of logs. A large share of log volume exists so someone can count occurrences. That is a metric — vastly cheaper, faster to query, and retained longer.

4. Tiered retention. Hot and searchable for days, compressed object storage for months, and accept that querying older data is slow. Almost all queries touch recent data.

5. Attribute cost to teams. Log spend charged to the service that produces it, visible in that team's own dashboard. Nothing reduces volume faster than a team seeing its own number.

6. Make deletion routine. A quarterly review of the highest-volume log lines, asking whether each was used in the last incident. Most were not.

The reframing

Logs are the most expensive telemetry per unit of insight, because they are unstructured, verbose and queried rarely. Metrics are cheap and aggregate; traces are sampled and give causality. Logs should carry what neither can: the specific detail of a specific event you are investigating.

A platform whose observability strategy is "log everything and search later" has chosen the most expensive possible approach — and typically discovers this when the telemetry bill exceeds the infrastructure bill.