Log Management

Definition

Log management is everything between an application emitting a line and someone querying it: collection, buffering, enrichment, routing, indexing, retention and access control.

The architecture that holds up

• Applications write to standard output or a local file. Never synchronously over the network to a remote system — a logging outage must not become an application outage, and a slow log sink must not add latency to requests.
• A local agent collects, buffers and ships. Buffering matters: a network blip should delay logs, not lose them.
• The pipeline enriches with service, version, environment, region and host, so applications do not have to.
• Routing by class. Not all logs deserve the same treatment: audit logs need long retention and tamper resistance; debug logs need days; security logs may need a separate, restricted destination.
• Tiered storage. Hot and searchable for days or weeks; cold object storage, queryable more slowly, for months or years.

Retention, which is where the requirements conflict

Three pressures pull in different directions and must be reconciled explicitly:

• Operational — days to weeks is usually enough for incident investigation.
• Compliance — audit logs may require years, with integrity guarantees.
• Privacy — personal data in logs is subject to deletion rights and minimisation obligations, which argues for the shortest retention possible.

The resolution is different retention per log class, which requires classifying logs at emission rather than deciding at the storage layer where everything looks the same.

Access control, frequently overlooked

Logs contain far more than people assume. Access should be role-based, audited, and personal data should be redacted or tokenised in the pipeline rather than relying on nobody looking. In practice logs are one of the least controlled and most sensitive datastores in most organisations.

Cost control

Log volume grows with traffic, so it is one of the few costs that scales exactly with success. In order of effectiveness: log less at the source, sample successful paths while keeping all errors, tier retention aggressively, and move anything that is really a counter into metrics.

Failure scenarios

• Synchronous remote logging on the request path.
• No buffering, so a collector restart loses logs — usually during the incident when they matter.
• Everything indexed, when most logs are never queried and could be archived cheaply.
• Personal data logged, discovered during an audit.
• Uniform retention, so audit requirements force expensive long retention for debug noise.

Interview question

"Design a log pipeline for a fleet where audit logs must be kept for seven years and debug logs for five days."