Storage Tiering Service  ·  View 15 of 31  ·  4 · Data

Packs

How objects below 1 MB become one immutable, self-describing unit, and what deletion costs afterwards.

Editable source SVG draw.io All views
Select Cohort tenant · folder · age Size gate members under 1 MB Build Pack builder one tenant · one key Entry format header · bytes · CRC Seal Read-back verify every entry Manifest footer + sidecar Commit Catalogue batch pack_id · offset Live with it Dead-space ledger live vs dead bytes Compactor dead ≥ 50% live members Packs — How Small Objects Become One Immutable Unit Journey / task Decision point Application we own Security / platform Data store batch A pack is never modified. A delete adds dead bytes; only compaction reclaims them, and compaction reads every live member back. v 1.0 · owner Storage Platform Architecture · date 2026-09

Decisions

  • One tenant and one encryption key per pack. Tenant offboarding, crypto-shredding and a read of one tenant's object never involve another tenant's bytes (ADR-08).
  • Cohorts are cut by tenant, then folder, then upload time, so objects that arrived together and are opened together sit next to each other in the pack.
  • Every entry carries its own header: tenant, object id, version, length, key id and CRC32C. The footer indexes the entries and a copy of it is stored as a sidecar manifest. A pack explains itself without the catalogue.

Numbers

  • Packs seal at 2 GB or after 7 days open. Compaction runs when dead bytes pass 50% of a pack and its members' minimum durations have expired.
  • Every sealed pack is read back in full and checked entry by entry before a single placement row is committed.

Risk

  • Compaction is a full read of everything still live in a pack: the most expensive operation the platform performs. Dead-space thresholds are set so that most packs are deleted whole when their cohort expires, rather than compacted.
◀  Access Telemetry All views One Classification Cycle  ▶
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises15-pack-anatomy