A globally distributed platform generates logs at every edge location. Should logs be centralised, kept local, or something else? Analyse the trade-off.

The tension

Centralising everything gives one place to query across the whole fleet, which is what an investigation needs. It costs an enormous amount of cross-region data transfer — frequently the dominant expense — plus a central ingestion tier sized for the aggregate of every location.

Keeping logs local is cheap and fast to write, and makes cross-location investigation nearly impossible: "which locations served this customer badly" requires querying hundreds of places.

The resolution: tier by aggregation level

1. Metrics aggregated at the edge and shipped centrally. Counters and histograms, pre-aggregated locally, so the central system has complete statistical coverage at tiny volume. This answers "is anything wrong and where" for a fraction of the cost of shipping logs.

2. Sampled traces shipped centrally, with tail-based selection keeping errors and slow requests. This answers "why" for the requests worth investigating.

3. Full logs retained locally, briefly, with the ability to query them centrally on demand. The critical capability: a central query that fans out to edge locations and returns matching lines, without shipping everything continuously. Investigations are rare; continuous shipping pays for every line whether or not it is read.

4. Errors and security-relevant events shipped centrally always, because these are low-volume and high-value, and because they must survive the loss of the location that produced them.

What makes this work

Consistent structure and identifiers everywhere, so a central query can target the right locations and correlate across them. Without a correlation identifier propagated globally, distributed query returns disconnected fragments.

Sampling rates recorded, so aggregate counts derived from sampled data remain accurate.

Local retention long enough to cover the detection delay. If a problem is detected six hours later and edge logs are kept for four, the evidence is gone. This is the parameter most often set too short, and its correctness depends on detection latency rather than on storage cost.

The general principle

Ship what is small and always needed; keep what is large and rarely read, close to where it is produced. The mistake is treating logs as one uniform category with one policy — the volume and value distribution is extremely skewed, and a single decision either overpays for the bulk or loses the important minority.