Storage Tiering Service  ·  View 14 of 31  ·  4 · Data

Access Telemetry

Reducing 1.4 million reads a second to the signal a classifier needs, and making sure a silent pipeline cannot pass for a quiet corpus.

Editable source SVG draw.io All views
Emit Placement resolver 1.4 M reads/s peak Staging reads rehydrated bytes Reduce at source Hourly dedup object · class · hour Canary event every 60 s Log Access topic Kafka · 7 d Store Raw events ClickHouse · 90 d Aggregates object · cohort · 3 y Use Classifier per cycle Freshness gate canary age ≤ 15 min Watch Healthchecks outside the platform Access Telemetry — From 1.4 M Reads a Second to One Signal Interface / broker Application we own Decision point Security / platform Queue / topic Data store Non-promoting classes are still recorded, for cost attribution, and are excluded from the promoting aggregate. v 1.0 · owner Storage Platform Architecture · date 2026-09

Decisions

  • The resolver emits the first read of each object by each reader class in each hour, with a count, instead of every read. Classification needs last access and distinct access days, not a range-read log (ADR-13).
  • Objects under 1 MB are aggregated per folder cohort at ingest, which is the unit they will be classified and moved as.
  • A canary event per Kafka partition every 60 seconds. The freshness gate and the external dead-man monitor both read its age, so the alarm and the brake can never disagree.

Numbers

  • Estimated 60,000 to 90,000 events a second at peak after deduplication, about 120 billion a month at 180 bytes each: roughly 22 TB raw a month, 3 to 5 TB compressed in ClickHouse.
  • Raw events kept 90 days, aggregates three years. Demotion stops when the newest canary is more than 15 minutes old.

Rejected

  • Flink or Spark streaming for aggregation. ClickHouse materialised views over a Kafka engine table do the rollup at ingest, and one fewer distributed system is one fewer thing to be the dominant controllable cost.
◀  Catalogue, Records and Policy: Data Model All views Packs  ▶
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises14-access-telemetry-pipeline