Storage Tiering Service  ·  View 02 of 31  ·  1 · Context and scope

High-Level Architecture

Six stages from an access event to a stored byte, and the one line down the middle that the read path never crosses.

Editable source SVG draw.io All views
Observe Access events reader class · hourly Access aggregates Kafka → ClickHouse Decide Classifier window · break-even Guards holds · pins · dwell Move Ring controller Temporal Mover fleet copy · verify Commit Placement catalogue Vitess · 9.2 B rows Serve Placement resolver p99 ≤ 8 ms Recall jobs handle · ETA Store Tier ladder Ceph · tape access events Storage Tiering Service — High-Level Architecture Queue / topic Data store Application we own Security / platform Interface / broker External / third party event / async Observe, Decide and Move may all be down at once. Serve reads only the committed catalogue, so every object still resolves. v 1.0 · owner Storage Platform Architecture · date 2026-09

Decisions

  • The catalogue commit, not the destination write, is the moment a placement changes. Everything left of it is advisory and may be slow, wrong or down.
  • Serve reads only the committed catalogue. It never calls the classifier, never evaluates a policy and never probes a tier to see where an object went.
  • The loop closes through telemetry, not through a call. A read emits an event; the event changes a future decision; the decision never touches the read that produced it.

Numbers

  • Placement resolution 99.99% monthly, p50 2 ms and p99 8 ms. The tiering plane carries 99.5%, and nobody should flinch at that, because nothing reads through it.
  • Blended storage cost at most $0.0104 per GB-month against an all-hot $0.020: a net reduction of at least 45%.
  • Movement capacity 900 TB and 120 million objects a day, 4× for a policy backfill.

Risks

  • The catalogue is the one hard dependency. If it is slow, every product read is slow. That is why it gets the most availability engineering in the design and the fewest features.