Storage Tiering Service  ·  View 13 of 31  ·  4 · Data

Catalogue, Records and Policy: Data Model

Twelve entities across three stores, and the one join every commit has to make without leaving its shard.

Editable source SVG draw.io All views
hold_scope hold_id PK tenant_id FK scope workspace | folder | owner floor_recall_class rm_reference approved_by two principals tenant_guard tenant_id PK guard_epoch demotion_frozen bool ingest_policy retrieval_budget_id placement tenant_id · object_id · version PK tier · location pack_id · offset · length size · checksum · key_id decision_id · ladder_sha min_duration_until · dwell_until row_version pack pack_id PK tenant_id · key_id tier · location · manifest live_bytes · dead_bytes sealed_at · checksum policy_version policy_sha PK state shadow | dry-run | live | frozen ring 0.1 | 1 | 10 | 100 move_cap_per_window decision decision_id PK policy_sha FK inputs window · reads · class p_read · break_even_at confidence · verdict movement movement_id PK object key FK from_location · to_location state copying | committed | released | abandoned guard_epoch_seen release_after ladder_version ladder_sha PK tiers ordered price · retrieval · min_duration latency · recall_class budget budget_id PK tenant_id · period ceiling · spent override_authority recall_job job_id PK budget_id FK authority tenant | legal state · eta staging_ttl recall_item job_id · object key PK pack_id · cartridge state · staged_at read_before_expiry bool movement_outcome movement_id PK FK bytes · requests charges $ elapsed_ms · result 1 : N 1 : N N : 1 1 : N N : 1 N : 1 N : 1 1 : 1 1 : N 1 : N Catalogue, Records and Policy — Data Model Top row lives in Vitess, one shard per tenant range. decision and movement_outcome are ClickHouse and Parquet; policy, budget and recall rows are PostgreSQL. v 1.0 · owner Storage Platform Architecture · date 2026-09

Decisions

  • tenant_guard sits in the same Vitess shard as the tenant's placements. A commit reads guard_epoch and writes the placement in one single-shard transaction; no transaction ever spans tenants (ADR-21).
  • Holds are scopes (workspace, folder, owner), not flags on nine billion rows. Placing a hold writes one scope row and bumps one epoch.
  • Pack membership is a column of placement (pack_id, offset, length). A pack is not a container objects belong to; it is a location.

Sizing

  • placement is under 400 bytes a row: about 3.7 TB for 9.2 billion objects, roughly 8 TB with secondary indexes, spread across 64 shards of 130 GB each.
  • movement rows exist only while a movement is in flight or awaiting release, so the table stays in the low hundreds of millions of rows.

Key design

  • Rows are sharded with a Vitess multicolumn vindex: a tenant hash prefix and an object hash suffix. A tenant with 400 million objects spreads across shards while single-tenant queries still route to a bounded subset.