Observability Platform  ·  View 14 of 25  ·  Data

Retention Tiers

Six stages from written to gone, at declining resolution and rising latency.

Editable source SVG draw.io All views
Written Metrics 10 s resolution Traces kept whole Logs by declared class Hot — interactive 10 s · 15 days p95 ≤ 2 s panels Traces · 14 days lookup p95 ≤ 1.5 s Logs · 7 days search p95 ≤ 15 s Warm — slower, cheaper 1 min · 90 days mergeable sketches Exemplar-linked · 90 days pivot targets only Logs · 30 days S3 Standard Cold — rehydrate to read 5 min · 13 months capacity planning Logs · 400 days Glacier IR, ≤ 15 min Long and locked 1 hour · 3 years trend only Compliance class · 7 years S3 Object Lock Gone Lifecycle expiry per class, automatic Subject erasure all tiers, ≤ 30 days Crypto-shred last resort downsample downsample age out reachable Retention Tiers — What Survives, At What Resolution, For How Long Data store Security / platform Application we own batch failure / alternate Percentile-bearing metrics are stored as mergeable sketches, never as pre-computed percentiles, so a tier change never silently changes what a chart means. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Percentile-bearing metrics are stored as mergeable sketches, never as pre-computed percentiles, so re-aggregation across a tier boundary is still correct and a chart cannot silently change meaning as it ages.
  • The resolution of any returned series is stated in the response. A 5-minute point and a 10-second point look identical on a chart and mean different things.
  • The cold tier is rehydrated rather than queried in place. It is explicitly not an incident tool — 15 minutes is useful to an auditor and useless at 03:14.

Numbers (assumptions)

  • Metrics: 10 s for 15 days, 1 min for 90 days, 5 min for 13 months, 1 hour for 3 years.
  • Logs: hot 7 days, warm 30 days, cold 400 days; compliance class 7 years under Object Lock. Traces: 14 days, exemplar-linked 90 days.
  • Every one of these is invented to be defensible and arguable. The 400-day log figure is the weakest and the most expensive: it exists to cover a year-over-year comparison nobody has yet asked for.

Risks

  • Cold-with-rehydration preserves questions nobody anticipated at a storage cost that grows forever. Downsample-and-discard is bounded and forecloses them permanently. This set commits to the first and leaves the bet visible in ADR-15.
◀  Data Model All views Critical Flow — Sample to Page  ▶
Solution Architecture v1.0 · Amazon Web Services with open-source collection and storage · Reliability Architecture · 2026-09 · 25 views14-retention-and-tiers