Storage Tiering Service

Architecture Views

31 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

A placement service for 9.2 billion objects held by a file-collaboration product, built from open-source software in two data centres the organisation owns: Vitess on MySQL for the placement catalogue, Ceph RGW for the hot, warm and cold tiers, EOS and the CERN Tape Archive for tape, Kafka and ClickHouse for access telemetry, Temporal for policy rollout and recall jobs, and SPIRE, OpenBao and Keycloak for identity and keys. Read the set in order. Acts 1 and 2 fix the boundary and the people who depend on it. Act 3 shows the parts, the tier ladder and the path of one read. Act 4 separates four kinds of state and shows how small objects become packs. Act 5 walks a classification cycle, the break-even arithmetic, one movement, a hold arriving mid-move, a tape recall and a retrieval storm. Acts 6 and 7 cover how it runs, what it costs, how it is kept honest and what fails. One boundary governs every page: the tiering plane decides where an object should be, the catalogue records where it is, and no read ever asks the tiering plane anything.

1 · Context and scope

What the service decides, what it reads through, what it places into, and the shared services it uses but does not own.
01 Who reads through it File service web · mobile · sync ML training pipelines non-promoting People and records authority Storage economics lead Legal operations Storage SRE Records management holds · retention Storage and its ledger Ceph object tiers hot · warm · cold Tape archive EOS + CTA · LTO Offsite vault Phase 3 custodian Finance chargeback storage ledger Storage Tiering Service 9.2 B objects · 41 PB Shared services and bulk readers Export and eDiscovery bulk readers Identity and keys Keycloak · SPIRE · OpenBao resolve placement declared reads policy · ladder holds · recalls drift · reversal hold events copy · verify · release archive · stage cartridges net saving recall jobs OIDC · SVIDs · keys Storage Tiering Service — System Context External / third party Person or role Security / platform synchronous event / async batch v 1.0 · owner Storage Platform Architecture · date 2026-09 System Context Who reads through the service, who changes what it does, the storage it places into, and the shared services it depends on without owning. HTML page SVG draw.io
02 Observe Access events reader class · hourly Access aggregates Kafka → ClickHouse Decide Classifier window · break-even Guards holds · pins · dwell Move Ring controller Temporal Mover fleet copy · verify Commit Placement catalogue Vitess · 9.2 B rows Serve Placement resolver p99 ≤ 8 ms Recall jobs handle · ETA Store Tier ladder Ceph · tape access events Storage Tiering Service — High-Level Architecture Queue / topic Data store Application we own Security / platform Interface / broker External / third party event / async Observe, Decide and Move may all be down at once. Serve reads only the committed catalogue, so every object still resolves. v 1.0 · owner Storage Platform Architecture · date 2026-09 High-Level Architecture Six stages from an access event to a stored byte, and the one line down the middle that the read path never crosses. HTML page SVG draw.io

2 · People and journeys

Who depends on placement, what each of them needs from it, and the three moments where a tiering platform either pays for itself or becomes an incident.
03 People who own the economics Storage economics lead FinOps Goal — Cut the storage bill by 45% and prove it net of every charge tiering causes. Core journeys Roll out a demotion policy Publish the net saving Score a classifier version Product engineer file service team Goal — Read any object with one call and never learn which tier it is in. Core journeys Resolve a placement Declare ingest placement Show a fetching state People who answer for the data Legal operations litigation support Goal — Produce a custodian's files by the court date without an open-ended bill. Core journeys Recall a custodian's archive Place a legal hold Security officer CISO org Goal — Know that the fleet which moves objects can never be the fleet that deletes them. Core journeys Approve a hold removal Audit a bulk recall People who run it Storage SRE on-call Goal — Undo a bad policy for what it cost, not for what it moved. Core journeys Reverse a classifier regression Rebuild the catalogue by scan Storage platform team Ceph and tape Goal — Change hardware and tiers without re-tiering nine billion objects. Core journeys Add a tier to the ladder Retire a Ceph cluster Machines with obligations ML training pipeline whole-corpus reads Goal — Read everything once without making any of it look hot. Core journeys Scan as a non-promoting reader Records management owns holds Goal — Place a hold and know nothing under it drops below its floor. Core journeys Publish hold changes Reconciler notifications + scan Goal — Find every byte the catalogue does not name, and say why. Core journeys Classify drift Actors and Their Core Journeys Person or role Journey / task External / third party Security / platform v 1.0 · owner Storage Platform Architecture · date 2026-09 Actors and Their Core Journeys The people and machines who depend on placement, what each is trying to get done, and the journeys that carry the value. HTML page SVG draw.io
04 Storage economics lead policy v42 Goal — Move documents idle for 90 days to cold, net positive Trigger — Quarterly bill 11% over plan; warm tier at 81% Done when — Full rollout, realised delta within 10% of shadow 1 · Draft 2 · Shadow 3 · Dry run 4 · First ring ◆ moment of truth 5 · Widen 6 · Score What they do Writes policy in CUE Runs shadow on corpus Dry run on 1% sample Approves 0.1% ring Reads realised delta Scores at day 90 System behind it Policy repo Shadow runner Break-even evaluator Ring controller Cost ledger Decision records How it feels Confident Unsure Anxious Where it hurts Which window per class? Projects 38%, not 45% Re-read rate 4.1% Early deletes creeping What the platform gives Ladder does the arithmetic Bill delta, 0 bytes moved Losing objects named Ring halts itself Revert free for 24 h Predicted vs realised Journey — Economics Lead Demotes Idle Documents to Cold v 1.0 · owner Storage Platform Architecture · date 2026-09 Journey: Demote Idle Documents to Cold The economics lead takes a policy from a merge request to every eligible object, and learns at the first ring that the corpus disagrees with the model. HTML page SVG draw.io
05 Legal operations litigation support Goal — 1.8 M files for one custodian in the review tool by Friday Trigger — Court order with a ten-day production deadline Done when — Every object staged, every charge attributed 1 · Request 2 · Authorise ◆ moment of truth 3 · Plan 4 · Wait ◆ moment of truth 5 · Review 6 · Close What they do Names custodian + range Cites the court order Reads ETA per tier Watches progress Exports to review tool Releases staging System behind it Recall API Budget authority Quota planner Tape library Staging bucket Charge ledger How it feels Calm Watchful Worried Where it hurts Tenant budget would refuse 31 cartridges, 6 drives ETA slips by 5 h 48 h TTL too short What the platform gives One job, not 1.8 M reads Legal authority, logged Grouped by pack and tape ETA republished honestly TTL extended per job Unread bytes reported Journey — Legal Operations Recalls Seven Years of One Custodian v 1.0 · owner Storage Platform Architecture · date 2026-09 Journey: Recall Seven Years of One Custodian Legal operations needs 1.8 million files by a court date. 700,000 of them are on tape, and the tenant's retrieval budget says no. HTML page SVG draw.io
06 Storage SRE on-call Goal — Stop policy v43 and put back only what it moved Trigger — Wrong-tier read rate alert at 03:10 Done when — Population restored; incident cost stated in dollars 1 · Alert 2 · Contain ◆ moment of truth 3 · Identify 4 · Reverse ◆ moment of truth 5 · Account 6 · Learn What they do Paged on wrong-tier rate Freezes policy v43 Lists moved population Starts reversal job Reads incident cost Adds a shadow check System behind it Alertmanager Ring controller Decision records Planner · reversal Cost ledger Policy repo How it feels In control Uneasy Under pressure Where it hurts 3.1 M objects moved Are rings still rolling? Some sources released Early-delete charges What the platform gives Ring stopped at 10% One flag halts all rings Population by version Under 24 h: pointer flip Charges per decision Regression becomes a gate Journey — Storage SRE Reverses a Bad Policy at 03:10 v 1.0 · owner Storage Platform Architecture · date 2026-09 Journey: Reverse a Bad Policy at 03:10 Policy v43 is sending a population to tape that is still being read. On-call has to stop it, find exactly what it moved and put it back without causing a second incident. HTML page SVG draw.io

3 · Structure

The layers, the containers, the declared tier ladder, every interface, and exactly what happens between a read request and its first byte.
07 Callers File service reads · uploads Export · eDiscovery bulk recall ML pipelines non-promoting Access and recall Placement resolver Go · bounded cache Read router ranges · pack offsets Placement API register · tombstone Recall service jobs · budgets Tiering control Classifier ClickHouse SQL Break-even evaluator Go · ladder maths Shadow runner 0 bytes moved Ring controller Temporal Movement Mover workers copy · verify · commit Pack builder seal · read back Budget authority $ and req/s leases Reconciler notify + scan Catalogue Placement catalogue Vitess · 64 shards Release gate delete identity Record and policy Event log Kafka · RF3 Analytics ClickHouse Decision records Parquet · Object Lock Policy and ladder Git · PostgreSQL Storage tiers Hot Ceph · NVMe Warm Ceph · HDD EC Cold · packed Ceph · dense HDD Archive · packed EOS + CTA · LTO resolve plans CAS commit Layered Architecture External / third party Interface / broker Application we own Decision point Security / platform Data store Queue / topic synchronous event / async No arrow from Access and recall reaches Tiering control or Movement. Remove both bands and every read still resolves. v 1.0 · owner Storage Platform Architecture · date 2026-09 Layered Architecture Seven layers in dependency order. Callers sit above what they call, and the access layer shares no arrow with the two layers that decide and move. HTML page SVG draw.io
08 Serving plane · Kubernetes · 99.99% Object access Placement resolver Go · 48 pods Read router Go library Placement API register · delete Catalogue vtgate Vitess router MySQL shards 64 × 3 · semi-sync Release gate Go · delete role Recall Recall API Go · job handles Recall workflows Temporal Unpack workers Go · staging Tiering plane · Kubernetes · 99.5% Observe Kafka events · decisions ClickHouse aggregates · replica Catalogue CDC VStream → Kafka Move Mover workers Go · 24 nodes Pack builder Go · compactor Reconciler Go Decide Classifier jobs SQL + Go Ring controller Temporal Policy service PostgreSQL Ceph RGW tiers hot · warm · cold EOS + CTA tape archive Records management hold events point read replica copy Container Architecture Interface / broker Application we own Data store Security / platform Queue / topic External / third party synchronous event / async The mover's CAS commit to vtgate and the recall path to tape are drawn on views 18 and 20. v 1.0 · owner Storage Platform Architecture · date 2026-09 Container Architecture Two Kubernetes deployments with different availability targets, different change cadences and different on-call expectations. HTML page SVG draw.io
09 Media · protection Storage $/GB-month Retrieval $/GB Minimum duration First byte p99 Recall class hot NVMe · 3× replica Ceph RGW $0.0200 the counterfactual $0 none 45 ms immediate warm HDD · EC 8+3 Ceph RGW $0.0080 $0.0005 30 days 120 ms immediate cold · packed 20 TB HDD · EC 12+4 packs below 1 MB $0.0042 $0.0030 90 days pack dead space 900 ms pack range read immediate archive · packed LTO-10 · 2 copies EOS + CTA $0.0011 $0.020 + mount $0.40 per mount 365 days repack cycle p50 4 min p99 45 min delayed deep archive · Phase 3 Exported LTO independent vault $0.0006 $0.080 + courier 365 days ≤ 72 h contract, not median scheduled The Tier Ladder — Declared Properties, Version 1 Prices are internal transfer prices from the storage team, fully loaded over five years. They are assumptions until the proof phase measures them. v 1.0 · owner Storage Platform Architecture · date 2026-09 The Tier Ladder Every placement calculation reads this table and nothing else. On-premises, prices and minimum durations are internal transfer prices, and each still has a physical cause. HTML page SVG draw.io
10 Callers File service Upload service Export and eDiscovery Platform Storage Tiering Service resolve · decide · move · recall Dependencies Ceph RGW tiers EOS + CTA Records management Resolve · gRPC Register Recall jobs S3 · STS Tape REST holds Integration Surface — Every Interface External / third party Application we own synchronous event / async Only Resolve and the GET after it are on a read path. ML pipelines call Resolve as non-promoting; identity, keys and the finance ledger are on view 01. v 1.0 · owner Storage Platform Architecture · date 2026-09 Integration Surface Three inbound interfaces, three outbound, and which of them a user's read actually waits on. HTML page SVG draw.io
11 Ask File service key · reader class Resolve Resolver cache TTL 60 s Catalogue replica lag cut-off 5 s Catalogue primary only on not-found Route Recall class from placement Read Read router pack offset → range Job handle ETA · within 3 s Bytes Ceph tier RGW GET · ranged delayed · scheduled Anatomy of a Read — Resolve, Route, Read External / third party Interface / broker Data store Decision point Application we own failure / alternate A source is released 24 h after commit: longer than the cache TTL, the replica lag and the longest stream. A stale answer still points at bytes. v 1.0 · owner Storage Platform Architecture · date 2026-09 Anatomy of a Read Everything between a file service asking for an object and the first byte, and why every shortcut on this path is safe. HTML page SVG draw.io

4 · Data

Four kinds of state with four recovery obligations, the catalogue model, how 1.4 million reads a second become one access signal, and how small objects become packs.
12 Catalogue · loss stops reads · RPO 0 Placement rows Vitess · 9.2 B Movement rows in-flight state Tenant guard rows hold epoch Telemetry · derived · RPO 5 min Raw access events ClickHouse · 90 d Access aggregates object · cohort · 3 y Canary heartbeats freshness Records · audit grade · RPO 60 s Decision records inputs · odds Movement outcomes bytes · requests · $ Parquet archive Object Lock · 3 y Policy · change-controlled Ladder versions Git · signed Policy versions CUE · rings Holds · pins · budgets PostgreSQL · dual auth Tier scan location back, history not Four Kinds of State, Four Recovery Obligations Data store Security / platform Risk / gap Each box can be lost without taking another with it. Only the first stops reads; the tier scan below rebuilds its locations, and only the third cannot be rebuilt at all. v 1.0 · owner Storage Platform Architecture · date 2026-09 Four Kinds of State The catalogue, the telemetry, the records and the policy, kept in separate stores because they fail, recover and matter differently. HTML page SVG draw.io
13 hold_scope hold_id PK tenant_id FK scope workspace | folder | owner floor_recall_class rm_reference approved_by two principals tenant_guard tenant_id PK guard_epoch demotion_frozen bool ingest_policy retrieval_budget_id placement tenant_id · object_id · version PK tier · location pack_id · offset · length size · checksum · key_id decision_id · ladder_sha min_duration_until · dwell_until row_version pack pack_id PK tenant_id · key_id tier · location · manifest live_bytes · dead_bytes sealed_at · checksum policy_version policy_sha PK state shadow | dry-run | live | frozen ring 0.1 | 1 | 10 | 100 move_cap_per_window decision decision_id PK policy_sha FK inputs window · reads · class p_read · break_even_at confidence · verdict movement movement_id PK object key FK from_location · to_location state copying | committed | released | abandoned guard_epoch_seen release_after ladder_version ladder_sha PK tiers ordered price · retrieval · min_duration latency · recall_class budget budget_id PK tenant_id · period ceiling · spent override_authority recall_job job_id PK budget_id FK authority tenant | legal state · eta staging_ttl recall_item job_id · object key PK pack_id · cartridge state · staged_at read_before_expiry bool movement_outcome movement_id PK FK bytes · requests charges $ elapsed_ms · result 1 : N 1 : N N : 1 1 : N N : 1 N : 1 N : 1 1 : 1 1 : N 1 : N Catalogue, Records and Policy — Data Model Top row lives in Vitess, one shard per tenant range. decision and movement_outcome are ClickHouse and Parquet; policy, budget and recall rows are PostgreSQL. v 1.0 · owner Storage Platform Architecture · date 2026-09 Catalogue, Records and Policy: Data Model Twelve entities across three stores, and the one join every commit has to make without leaving its shard. HTML page SVG draw.io
14 Emit Placement resolver 1.4 M reads/s peak Staging reads rehydrated bytes Reduce at source Hourly dedup object · class · hour Canary event every 60 s Log Access topic Kafka · 7 d Store Raw events ClickHouse · 90 d Aggregates object · cohort · 3 y Use Classifier per cycle Freshness gate canary age ≤ 15 min Watch Healthchecks outside the platform Access Telemetry — From 1.4 M Reads a Second to One Signal Interface / broker Application we own Decision point Security / platform Queue / topic Data store Non-promoting classes are still recorded, for cost attribution, and are excluded from the promoting aggregate. v 1.0 · owner Storage Platform Architecture · date 2026-09 Access Telemetry Reducing 1.4 million reads a second to the signal a classifier needs, and making sure a silent pipeline cannot pass for a quiet corpus. HTML page SVG draw.io
15 Select Cohort tenant · folder · age Size gate members under 1 MB Build Pack builder one tenant · one key Entry format header · bytes · CRC Seal Read-back verify every entry Manifest footer + sidecar Commit Catalogue batch pack_id · offset Live with it Dead-space ledger live vs dead bytes Compactor dead ≥ 50% live members Packs — How Small Objects Become One Immutable Unit Journey / task Decision point Application we own Security / platform Data store batch A pack is never modified. A delete adds dead bytes; only compaction reclaims them, and compaction reads every live member back. v 1.0 · owner Storage Platform Architecture · date 2026-09 Packs How objects below 1 MB become one immutable, self-describing unit, and what deletion costs afterwards. HTML page SVG draw.io

5 · Runtime

A classification cycle, the break-even inequality on real objects, one movement end to end, a hold and a delete arriving mid-move, a tape recall and a retrieval storm.
16 Candidate Window Break-even Guards Record Plan Object ≥ 1 MB Idle 97 days last promoting read 90 d window met Saving 40× costs No hold · dwell met Recorded with odds Batch to warm Small-object cohort Folder · 41k objects all under 1 MB Window per folder Priced as one pack Guards per member One decision · 41k Pack build Periodic class Quarterly report idle 120 days 400 d window not met Decision: stay Under legal hold Custodian exports idle 3 years Window met Archive pays 11× Floor: immediate Refused and alerted One Classification Cycle — Four Kinds of Candidate The cycle reads the analytical replica. Every guard is checked again, against the catalogue itself, at the moment of commit (view 19). v 1.0 · owner Storage Platform Architecture · date 2026-09 One Classification Cycle Four kinds of candidate go through the same six steps, and three of them end somewhere other than a movement. HTML page SVG draw.io
17 Saving over min duration Expected retrieval Move + early delete Verdict 8 GB video · hot → warm $0.096 30 d × $0.012/GB $0.0006 p(read) 15% $0.0017 p(delete) 2% Demote margin 40× 240 KB photo, alone · hot → cold $0.00001 ≈ $0 $0.00015 overhead share Refuse costs 13× saving 41k photos, one folder · hot → cold pack $0.46 9.8 GB × 90 d $0.0015 p(read) 5% $0.05 41k GETs · 1 pack Demote as a pack margin 9× 2 GB quarterly report · warm → archive $0.17 365 d × $0.0069/GB $1.76 4 recalls + mounts $0.40 mount share Refuse costs 13× saving The Break-Even Inequality — Four Worked Examples Demote only if saving over the destination's minimum duration exceeds p(read) × retrieval + movement + p(early delete) × early-delete charge. v 1.0 · owner Storage Platform Architecture · date 2026-09 The Break-Even Inequality The same arithmetic applied to four real kinds of object: two demote, two are refused, and the reasons are different. HTML page SVG draw.io
18 Mover worker Budget authority Warm tier · DC-A Cold tier · DC-A Catalogue Release gate Cold tier · DC-B 1. lease 40 GB · 5,000 requests 2. granted for 5 min 3. mark in-flight m-77 if row_version 12 4. ok · guard_epoch 9 5. GET object 6. PUT with CRC32C 7. stored · checksum matches 8. commit if epoch 9 and not tombstoned 9. committed · release_after +24 h 10. releases due on shard 17 11. HEAD destination replica 12. present · checksum ok 13. DELETE source copy 14. mark released One Movement — Copy, Verify, Commit, Release Until message 9 the object reads from warm. After it, reads go to cold while warm stays intact for 24 h as a free undo. v 1.0 · owner Storage Platform Architecture · date 2026-09 One Movement: Copy, Verify, Commit, Release The four steps in order, the two conditions a commit checks, and the three conditions a release checks, for one object going from warm to cold. HTML page SVG draw.io
19 Records management Policy service File service Catalogue Mover worker Release gate Destination tier 1. mark in-flight m-91 · epoch 9 2. PUT copy 3. hold H-204 · owner j.doe 4. restrictive · applies at once 5. hold scope + guard_epoch 10 6. commit m-91 if epoch 9 7. rejected · epoch is 10 8. tombstone object o-55 9. deleted · m-92 abandoned 10. commit m-92 11. rejected · tombstone 12. abandon m-91 · m-92 13. DELETE both destinations A Hold and a Delete Arrive While Objects Are Moving One epoch bump invalidates every uncommitted movement for the tenant. No object row is touched to place the hold. v 1.0 · owner Storage Platform Architecture · date 2026-09 A Hold and a Delete Arrive Mid-Move Two of the requirement's hardest rules, a hold forbids demotion and a delete wins, enforced by one transaction on one shard. HTML page SVG draw.io
20 Legal operations Recall API Catalogue Budget authority Recall workflow EOS + CTA Staging bucket 1. recall owner j.doe · 2019–2025 2. resolve 1.8 M placements 3. 1.1 M immediate · 0.7 M delayed 4. price 2,300 packs on 31 cartridges 5. over tenant ceiling · legal authority 6. job J-88 · ETA 31 h · approver needed 7. approved by counsel 8. start J-88 9. stage 2,300 packs · Tape REST 10. order by cartridge and position 11. packs staged, in batches 12. unpack members · TTL 7 d 13. ETA revised to 36 h 14. ready · unread bytes reported at expiry Recall From Tape — A Job, Not a Hung Request Staging is not promotion: the placement rows still say archive, and the staged copies expire unless a policy decides otherwise. v 1.0 · owner Storage Platform Architecture · date 2026-09 Recall From Tape A request for 1.8 million objects becomes one job with a price, an authority, an honest estimate and a report of what was never read. HTML page SVG draw.io
21 Arrives Admitted by Planned as Draws on Staged for Ends Interactive reads 0.7% of reads non-immediate Job handle ≤ 3 s Single object 2 drives reserved 48 h Promotion by policy eDiscovery · legal 0.7 M archived objects Legal authority never refused Packs by cartridge Up to 8 drives 7 days Unread bytes charged Tenant migration export 42 M objects Tenant budget hard ceiling Queued behind legal Remaining drives 48 h Paused at ceiling resumable ML training scan Whole corpus non-promoting Immediate tiers only Archive excluded Ceph · 2 Gbps cap No staging No placement change Retrieval Storm — Four Demands on Sixteen Tape Drives The storm degrades to a longer published ETA, never to a larger bill. Drive reservations are ladder configuration, not code. v 1.0 · owner Storage Platform Architecture · date 2026-09 Retrieval Storm Four simultaneous demands on the same sixteen tape drives, and how each is admitted, planned and bounded. HTML page SVG draw.io

6 · Operations

Where it runs, how a policy reaches every object, how each placement is scored, what is watched, how the saving is counted, and the build order.
22 DC-A · primary · three rooms Serving and tiering Kubernetes clusters serving · tiering Vitess primaries 64 shards · semi-sync Kafka · ClickHouse 6 + 12 nodes Storage tiers Ceph hot · warm · cold RGW multisite zone A Tape library A EOS buffer · CTA DC-B · secondary Standby Kubernetes serving warm · resolvers up Vitess replicas async · RPO ≤ 10 s Storage tiers Ceph hot · warm · cold RGW multisite zone B Tape library B second copy Witness site etcd voter Vitess topology Healthchecks dead-man monitor Offsite vault Phase 3 · cartridges binlog multisite sync Deployment — Two Data Centres and a Witness Security / platform Data store Queue / topic External / third party event / async In-DC failover of a shard primary is VTOrc, under 60 s. Moving serving to DC-B is a declared runbook with a 15 min RTO. v 1.0 · owner Infrastructure · date 2026-09 Deployment Two data centres with three fault-isolated rooms in the primary, a witness for quorum, and storage that replicates independently of the service. HTML page SVG draw.io
23 Author Policy merge request CUE · Forgejo Two reviewers economics + SRE Validate Schema + invariants no hard-coded prices Signed bundle cosign Shadow Whole corpus 0 bytes moved Projected bill delta vs live policy Dry run 1% stratified sample copies, never commits Rings 0.1 → 1 → 10 → 100% Temporal Ring gate re-reads · charges Freeze + revert pointer flip within 24 h Score Predicted vs realised day 30 · day 90 gate fails Policy Delivery — From Merge Request to 100% of Objects Data store Person or role Security / platform Application we own Decision point Risk / gap failure / alternate No policy moves more than 2% of objects or 1.5 PB in any 7-day window, whatever its rings say. v 1.0 · owner Storage Platform Architecture · date 2026-09 Policy Delivery The six stages a policy passes before it has moved every object it is allowed to move, and the one condition that sends it back. HTML page SVG draw.io
24 Observe access aggregates Decide break-even recorded Move copy · verify · commit Dwell ≥ 14 days Measure reads · charges Score predicted vs realised Tune new version · shadow Placement bet window elapsed decision id committed dwell over day 30 · day 90 per version shadow passes A Placement Is a Bet — Made, Held, Scored, Improved Application we own Decision point Security / platform Data store Opportunity v 1.0 · owner Storage Platform Architecture · date 2026-09 The Placement Bet Lifecycle How every placement is made, held, measured against the prediction that justified it, and fed back into the next policy version. HTML page SVG draw.io
25 Metrics Records and logs Traces Alert and action Placement resolution p99 · cache hit replica lag Not found at location Slow resolves 99.99% burn · page Telemetry pipeline Canary age consumer lag Dropped events Age over 15 min demotion stops Movement $ and req/s vs lease Movement outcomes Per batch Budget 90% · ticket Recall Queue · drives busy published ETA Job records Per job ETA past ceiling · page Economics Wrong-tier rate charge ratio Decision records Charges over 12% demotion paused Drift Drift by cause Reconciler findings Unexplained · ticket Observability — What Is Watched, and What Stops When It Moves Prometheus and Thanos for metrics, ClickHouse for records, OpenTelemetry to Tempo for traces. The canary is also watched from the witness site. v 1.0 · owner SRE · date 2026-09 Observability Six things that are watched, what records them, and the action each alert takes, several of which stop the platform rather than page a person. HTML page SVG draw.io
26 Inputs Placed bytes catalogue replica · daily Movement outcomes bytes · requests Recall charges retrieval · mounts Platform cost hardware · power share Price Ladder version price at placement Compute Counterfactual bill all hot · $0.020 Realised bill by tier Tiering charges retrieval · early delete Attribute Per decision tenant · class · tier Publish Net saving target ≥ 45% Platform cost ratio target ≤ 8% of net Chargeback check ± 2% of storage ledger The Cost Ledger — Net Saving Against an All-Hot Counterfactual Data store Decision point Application we own Opportunity Security / platform A saving with no counterfactual is an assertion, and a gross saving is not a saving. Both numbers are published side by side every month. v 1.0 · owner FinOps · date 2026-09 The Cost Ledger How the saving is computed so that it survives a finance review: against a counterfactual, net of every charge, attributed to decisions, and reconciled. HTML page SVG draw.io
27 Proof · 10 weeks Resolver at 1.4 M/s p99 measured Movement under chaos kill · drop commits Shadow on real logs projected saving MVP · to month 6 Catalogue on every read Hot · warm · cold objects ≥ 1 MB Declarative policy break-even · rings Holds · budgets · drift Phase 2 · months 6–12 Packs + compaction cohorts under 1 MB Tape archive EOS + CTA Bulk recall · staging Predictive, in shadow Phase 3 · months 12–24 Offsite vault custodian Recall price quotes Tenant cost reporting Region as a dimension Roadmap — Never Lose an Object, Then Save Money, Then Go Deeper Application we own Decision point Security / platform The MVP leaves objects under 1 MB where they are. That forgoes about 7% of the target saving and avoids 75% of the movement requests. v 1.0 · owner Storage Platform Architecture · date 2026-09 Roadmap Prove that nothing is lost, then earn the saving on large objects, then go after small objects, tape and a second custodian. HTML page SVG draw.io

7 · Assurance

Trust zones, credentials that expire in minutes, how disagreement between catalogue and storage is settled, and every named failure with what contains it.
28 Product network File service tenant token Serving plane Placement resolver catalogue read-only Recall API per-tenant authz Tiering plane Mover workers PUT-only · 15 min Compromised mover can copy, not delete Release Release gate DELETE if committed Data Catalogue grants per role Ceph RGW STS roles · SSE-KMS Custody OpenBao tenant keys Decision records Object Lock · 3 y mTLS PUT only DELETE denied DELETE source key id kept Security — Trust Zones and What Each Identity May Do External / third party Interface / broker Application we own Risk / gap Security / platform Data store synchronous failure / alternate No principal holds both a movement role and a delete role. Removing a hold or pin needs two named approvers and is written to the locked record first. v 1.0 · owner Security Architecture · date 2026-09 Security: Trust Zones Six zones ordered by exposure, the identity each crossing uses, and why a compromised mover can copy data but cannot destroy it. HTML page SVG draw.io
29 Mover pod SPIRE agent SPIRE server RGW STS Ceph RGW OpenBao Release gate 1. workload API · attest pod 2. node and pod selectors 3. JWT-SVID · aud rgw · 5 min 4. AssumeRoleWithWebIdentity 5. verify via SPIRE OIDC discovery 6. 15 min · PutObject on cold 7. PUT · SSE-KMS key of tenant 8. wrap data key 9. DELETE source object 10. AccessDenied 11. assume release-warm role 12. 15 min · DeleteObject on warm Short-Lived Credentials — One Tier, One Operation, Fifteen Minutes Nothing in the mover fleet is a long-lived secret. A stolen token is worth at most fifteen minutes of copying into one tier. v 1.0 · owner Security Architecture · date 2026-09 Short-Lived Credentials How a mover gets permission to write into exactly one tier for fifteen minutes, with no stored secret anywhere in the fleet. HTML page SVG draw.io
30 Found by Believed for where bytes are Action Recorded as Copy exists, commit lost Stuck in-flight row Tier Adopt if checksum matches Expected · paid twice Bytes with no row Rolling tier scan Tier Quarantine 35 d then gate deletes Orphan Class changed out of band Bucket notification Tier Correct row · revert Governance finding Row names missing bytes Read 404 · scan Tier Search replicas · mark unreadable Durability incident Checksum mismatch Sampled scrub Neither alone Repair from DC-B copy Durability incident Reconciliation — Every Kind of Disagreement, and Who Is Believed The tier is believed for where the bytes are; the catalogue for where they should be. Drift nobody can explain is a defect, never a reconciliation outcome. v 1.0 · owner SRE · date 2026-09 Reconciliation and Drift Five ways the catalogue and storage can disagree, which one is believed about what, and what each disagreement becomes. HTML page SVG draw.io
31 Detected by Contained by Reads see Back to normal Mover lost mid-copy Lease expiry No catalogue change Nothing Restart · partial deleted Telemetry stalls Canary age · Healthchecks Demotion stops Nothing Two fresh cycles Classifier regression Ring gate wrong-tier rate Ring · 2% window cap Some slower reads Pointer flip within 24 h Retrieval storm Queue depth Budgets · drive reservations Longer ETA Queue drains Shard primary lost VTOrc Semi-sync replica promoted ≤ 60 s errors, one shard Automatic DC-A lost Site monitoring DC-B runbook ≤ 15 min outage Sources still held 24 h Catalogue corrupted Reconciler · checksums Tier scan rebuild Errors in affected range ≤ 30 h · history lost Budget ends mid-recall Budget authority Job paused · staged kept Longer ETA Approver resumes Pack members lost Ceph unfound objects Members marked unreadable Explicit error never just slow Storage incident Failure Modes — Detection, Containment, What Reads See With classifier, planner and movers all down, every object reads at its current placement and every recall in flight completes. v 1.0 · owner SRE · date 2026-09 Failure Modes Nine named failures, how each is detected, what contains it, what a read sees, and how the system returns to normal. HTML page SVG draw.io

Architecture One-Pager

The problem, the shape of the answer, the decisions that carry it, and why it should still be the right shape in ten years.

Every demotion is a bet that an object will not be needed before the saving pays for the move. This architecture makes each bet explicit and priced, keeps every bet off the read path, and makes sure that a losing bet costs money and never costs an object.

A file-collaboration product keeps everything its customers ever upload: 9.2 billion objects and 41 PB today, growing 34% a year. Most of it is read heavily for a few weeks and then almost never, but almost never is not never, and nobody can say in advance which old file a customer will open tomorrow. Left alone, the storage bill grows with everything ever stored. Tiered carelessly, it grows anyway, through retrieval charges, early-deletion charges, movement costs and a telemetry pipeline that can cost more than it saves, while reads that used to take 40 milliseconds start taking four minutes. The hard parts are not the storage tiers. They are knowing when absence of access really means coldness, pricing each move honestly, moving nine billion objects without ever holding the only copy somewhere unrecorded, and making sure that the machinery doing all of this can fail completely without a single read failing with it.

Every read resolves its object through a placement resolver: a stateless Go service with a 60-second cache in front of a Vitess catalogue on MySQL, sharded by tenant. The resolver returns a location, a byte range and a recall class, and the file service reads the bytes directly from Ceph RGW. Hot, warm and cold tiers are separate Ceph clusters; objects under 1 MB are stored in cold as single-tenant packs; archive is LTO tape behind EOS and the CERN Tape Archive. Each resolved read emits a deduplicated access event into Kafka, and ClickHouse keeps per-object and per-cohort aggregates. Every six hours a classification cycle reads those aggregates and a change-data replica of the catalogue, evaluates each candidate's observation window and break-even against the signed tier ladder, and records every decision with its inputs. A Temporal workflow rolls a policy out through shadow, dry run and rings. Go movers copy each object, verify its checksum, and commit the new placement with a compare-and-set that fails if a hold arrived or the object was deleted. Twenty-four hours later a separate release gate, holding the only delete permission, removes the source once the destination is confirmed in the second data centre. Recalls from tape are Temporal jobs planned by pack and cartridge, charged against budgets and staged with a time limit. Every charge is attributed to the decision that caused it and reported against what the corpus would have cost if it had all stayed hot.

What it is, and what it is not

A catalogue lookup on every reada read path that asks a classifier, a policy engine or a storage tier where an object went.
Placement decisions recorded with their odds and their pricelifecycle rules that move objects nobody can later account for.
A saving counted net, against an all-hot counterfactuala gross storage delta that leaves out retrieval, early deletion, movement and the platform's own cost.
Recall classes with published estimatestier names, vendor classes or hung requests leaking into the product.
A service that places objects into storage it does not runa storage system, a deletion policy or an encryption scheme.
Tiering for objects that pay for ita platform that moves 240 KB photos one at a time at a loss.

The decisions that are the architecture

01The read path never asks the tiering plane

Resolution is a catalogue lookup and nothing else. The classifier, planner and movers carry 99.5% availability; placement resolution carries 99.99%, and the first can be switched off without touching the second.

ADR-01

02Stale answers are safe because sources outlive them

A source copy is released 24 hours after commit, which is longer than any cache, replica lag or stream. Resolution can use caches and replicas, and every movement has a free undo for a day.

ADR-03

03A movement is a catalogue state machine

Copy, verify, commit, release, with the in-flight state in the catalogue row. A worker that dies anywhere leaves a row that says what to do next, and the source is never released before the commit.

ADR-23

04A hold is an epoch, checked inside the commit

Holds are scope rows beside the tenant's placements. Placing one bumps a per-tenant epoch, and every movement planned under the old epoch fails to commit. No row in nine billion needs touching.

ADR-21

05Every demotion clears a recorded break-even

Saving over the destination's minimum duration must exceed expected retrieval, movement and early-deletion cost. The inputs, the odds and the verdict are stored, so every decision can be scored.

ADR-19

06Small objects move as cohorts in packs

Below 1 MB the per-object overhead exceeds the saving. Those objects are classified by folder and moved only as members of a single-tenant pack, or not at all.

ADR-17

07Blind means no demotion

One canary per telemetry partition drives both the freshness gate and an external dead-man monitor. When the signal is stale, demotion stops and promotion, recall and reads carry on.

ADR-18

08Moving and deleting are different identities

Movers can only write into one tier with 15-minute credentials. Only the release gate can delete, and only a source whose destination is committed and replicated.

ADR-29

Why this holds up over time

Tiering platforms tend to fail slowly rather than suddenly. Prices and media change and the hard-coded policy quietly becomes a money-losing one. The classifier is replaced and nobody can say whether the new one is better. The object count grows until per-object telemetry and requests cost more than the tiers save. The storage product or the tape generation changes and the platform turns out to depend on it. The engineers who knew why objects are where they are move on. Or a core component changes its licence. The design addresses each of these deliberately.

Prices are data, not code

The ladder is signed, versioned configuration, and every placement records the version it was made under. A new tape generation, a cheaper disk or a retired tier is a ladder change evaluated in shadow, not a code change, and existing placements stay valid under the version that made them.

Decisions carry their own evidence

Each decision stores its inputs, its predicted read probability and its break-even date, and each charge is attributed to one. A future team replacing the classifier compares the two on realised money, not on opinion, and does not need anyone from this team in the room.

Work grows with bytes, not with object count

Telemetry is deduplicated per object-hour, small objects are handled as cohorts, and packs turn millions of requests into one. Doubling the corpus to 18 billion mostly small objects adds catalogue rows but little telemetry and very few movements.

Storage sits behind standard interfaces

The S3 API and STS for disk tiers, the WLCG Tape REST API for tape. Ceph can be replaced by another S3 store and EOS with CTA by dCache without changing the placement model. LTO generations roll through CTA's repack, invisible to the catalogue except as a location change.

Every object can be found without this software

Object keys encode tenant, object and version, and every pack carries its own index and a sidecar manifest. If the catalogue and everyone who built it were gone, a script and the tiers are enough to recover where every object is.

Correctness comes from ordering, not from a product

Copy, verify, commit, release and an epoch-checked commit need only a store with conditional writes and an object store with PUT and DELETE. The safety argument survives a change of database and a change of storage.

Governance risk is named, with an exit for each

Vitess, Kubernetes, SPIRE and OpenBao sit under neutral foundations. Where one company leads, the exit is named: ClickHouse to Parquet with Trino, Temporal to PostgreSQL-backed workers, CTA to dCache or plain LTFS. CockroachDB and MinIO were considered and set aside after their licensing and community-edition changes showed what that risk looks like.

The platform measures itself

Net saving, platform cost as a share of it, per-object overhead and the smallest object worth tiering are published every month. Whoever re-decides these choices starts from those numbers rather than from this document.

Non-functional targets

Targets from the requirement, and the mechanism that meets each. Every latency, throughput and cost figure is replaced by a measured value in the proof phase and republished monthly.

QualityTargetHow it is metView
Placement resolution availability ≥ 99.99% monthly Stateless resolvers in three rooms; Vitess semi-sync with VTOrc failover; nothing from the tiering plane on the path 08
Placement resolution latency p50 ≤ 2 ms · p99 ≤ 8 ms 60 s in-process cache, lag-bounded replica reads, primary only for rows not yet replicated 11
First byte by tier p99 45 ms hot · 120 ms warm · 900 ms cold NVMe replicas, HDD erasure coding, ranged GET into cold packs without reconstructing the pack 09
Archive rehydration p50 ≤ 4 min · p99 ≤ 45 min · 12 h ceiling Two drives reserved for interactive recalls; CTA orders stages by cartridge position; estimates published per tier 21
Synchronous ceiling ≤ 3 s, then a job handle Recall class known at resolution; delayed and scheduled objects return a handle and never hold the request 11
Read throughput 1.4 M resolutions/s · 4× burst for 60 s 48 resolver pods sized on cache hits; catalogue replicas sized to survive a 50% cache hit rate 08
Movement capacity 900 TB and 120 M objects/day · 4× backfill 24 mover nodes on budget leases; small objects moved as packs; backfills yield to steady-state work 18
Bulk recall 50 M objects in one job Temporal job planned by pack and cartridge, quota-aware, resumable across budget pauses 20
Classification cycle ≤ 6 h over the whole corpus ClickHouse SQL over the CDC replica and aggregates; break-even arithmetic in Go on the candidate set only 16
Net storage cost ≤ $0.0104/GB-month · ≥ 45% below all-hot Break-even per decision, cohorts for small objects, ingest placement, net saving reconciled against chargeback 26
Tiering charges ≤ 6% of gross saving · demotion pauses at 12% Minimum duration as the bet horizon, 14-day dwell, 3× promotion bar, ring gates on charge ratio 23
Read quality Wrong-tier reads ≤ 0.05% · unread rehydration ≤ 3% Periodic classes with 400-day windows, drive reservations, staging TTL and unread-byte attribution 25
Object data RPO 0 Copy, verify, commit, release; source released 24 h after commit and only after the destination replica is confirmed 18
Catalogue recovery RPO 0 · RTO 60 s in DC · RPO 10 s · RTO 15 min across sites Semi-sync replicas in separate rooms; async replicas and warm resolvers in DC-B; rehearsed promotion runbook 22
Catalogue rebuild ≤ 30 h, location only Self-describing keys and pack manifests scanned in parallel; history restored from the records store 30

Scope

In scope

  • Placement resolution and the recall class contract for every object read
  • The placement catalogue, its analytical replica and its rebuild by tier scan
  • Access telemetry with reader classes, deduplication and a freshness gate
  • Declarative classification with per-object break-even, windows, dwell and promotion rules
  • Policy delivery through shadow, dry run, rings and a movement cap
  • Copy, verify, commit and release across Ceph tiers and tape, with packing and compaction
  • Recall jobs, staging, retrieval budgets and legal authority
  • Holds and pins as constraints, reconciliation and drift, and the net cost ledger

Explicitly out of scope

  • Operating Ceph and the tape estate, and the durability each tier declares
  • Deciding what may be deleted, retention schedules and determining legal holds
  • Serving bytes to end users, client and edge caching
  • Deduplication, compression and the choice of encryption at rest
  • Tenant-facing cost reporting and self-service policy, which are Phase 3

The ten-week proof

The proof does not try out features. It tests the properties everything else depends on: the read path at full load, a movement protocol that cannot lose an object, a saving that exists on real access logs, and the five product behaviours the design takes on trust.

  1. Resolver and a 16-shard Vitess catalogue loaded with 2 billion synthetic placements, driven at 1.4 M resolutions a second with a 4× burst
  2. Movers, release gate and reconciler against two small Ceph clusters in two rooms, with RGW STS and SPIRE credentials
  3. Shadow classification over 90 days of real access logs from the current product, producing a projected net saving
  4. One EOS and CTA instance with a small library, driven through the Tape REST API at 2,000 concurrent stage requests
  5. A VStream replica into ClickHouse kept current under 1,400 commits a second
  • Run 10 million movements while killing workers, dropping commit acknowledgements and restarting shard primaries; then prove by scan that every object is readable and exactly one copy is named
  • Place a hold on a tenant with 50,000 movements in flight; prove that none of them commits and none of their sources is released
  • Stop the telemetry pipeline; prove that demotion stops within 15 minutes, promotion and recall continue, and the external monitor pages
  • Cut DC-A in the middle of a release batch; prove that no object loses its last copy

Open risks, carried rather than hidden

RiskIf it landsResponse
The requirement's figures do not agree with each other 1.4 M reads a second does not fit 120 B events a month without deduplication; 210 TB a day of ingest does not fit 34% growth without heavy deletion; the per-object overhead target is thirteen times looser than the 8% cost ceiling Deduplicate at source (ADR-13), treat early deletion as a first-class input (ADR-19), and hold the platform to the tighter 8% figure, reported monthly
The platform costs more than 8% of the saving in year one Hardware alone is estimated at $450,000 to $600,000 a year against a ceiling of about $350,000 State it; the ratio is met in year two or three as the corpus grows against a mostly fixed platform. If the proof shows otherwise, buy less tiering: defer tape and packs (ADR-28)
Five product behaviours are unconfirmed STS with SPIRE, RGW checksums, notification reliability, Tape REST at volume or VStream at scale may not behave as assumed Each is a named proof test with a fallback: per-tier RGW keys rotated hourly from OpenBao, read-back verification, shorter scan interval, CTA native frontend, periodic snapshot export
The catalogue is on every product read A catalogue incident is a product outage, and its latency is a floor on every read Fewest features of any component, semi-sync across rooms, warm DC-B, 24-hour release hold so stale caches still serve; the catalogue gets the platform's strictest change process (ADR-02, ADR-03)
Objects under 1 MB stay hot through the MVP About 7% of the target saving is deferred to Phase 2 Accepted: they are 75% of objects and 4% of bytes, and moving them one at a time would lose money (ADR-17)
Tape skills and CTA experience are scarce outside research computing Slow incident response on the archive rung; hiring difficulty Archive is Phase 2, behind a standard API; the storage team runs it; LTFS and dCache are named alternatives if CTA does not fit (ADR-10)

Architecture Decision Record

Why every component and every technology on these 31 views is what it is, and what each choice costs.

Thirty-two decisions make up this architecture. Everything else on the thirty-one views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the products, the media and the team have changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Prices, latencies, volumes and savings are targets or stated assumptions taken from the requirement or from planning estimates, and each is replaced by a measured value in the proof phase. Five product behaviours are relied on and must be confirmed before production depends on them: Ceph RGW's STS accepting SPIRE-issued JWT-SVIDs through AssumeRoleWithWebIdentity; RGW validating S3 additional checksums (CRC32C) on PUT in the deployed Ceph release; RGW bucket notifications delivering object and lifecycle events to Kafka without silent loss under load; EOS and CTA serving 2,000 concurrent stage requests through the WLCG Tape REST API; and Vitess VStream keeping a 9.2-billion-row analytical replica current while the catalogue takes 1,400 commits a second.

How to read a record

QuestionThe forcing question: why a decision was needed at all.
ContextThe requirement, the scale and the constraint that make it hard.
DecisionWhat this architecture does, stated so it can be checked.
How it works on-premiseThe concrete mechanism: which package, configured how, on whose hardware.
Options weighedChosen, rejected, deferred, or right elsewhere, with the reason for each.
ConsequencesWhat the choice buys and what it costs, both kept visible.
Choose differently whenThe conditions that would flip the decision for your system.
Why it holds up over timeWhat keeps the decision right as scale, staff and technology change.
LessonThe principle that transfers beyond this platform.

Decision map

Read path and catalogue 5

What a read is allowed to depend on, where placement truth lives, and why stale answers are safe.

ADR-01The read path never consults the tiering plane ADR-02The placement catalogue runs on Vitess over MySQL, sharded by tenant ADR-03Bounded-stale resolution, made safe by a 24-hour release hold ADR-04The catalogue decides, the tier is fact, and both can be rebuilt from storage ADR-05Callers see a recall class and a job handle, never a tier

Tiers and storage 6

What a tier is when the organisation owns the hardware, which software provides each rung, and how small objects are stored.

ADR-06The tier ladder is signed, versioned configuration priced in internal transfer prices ADR-07Ceph RGW provides the disk tiers, one cluster set per tier, with lifecycle transitions disabled ADR-08Small objects are stored in single-tenant, single-key, self-describing packs ADR-09Compact a pack only when dead space passes half and minimum durations have expired ADR-10The archive rung is LTO tape behind EOS and CTA, driven through the WLCG Tape REST API ADR-11The ladder ends in a second custodian, in Phase 3, only for data kept as an obligation

Signal and classification 7

What counts as evidence of use, how it is collected cheaply, who classifies, and what happens when the signal goes quiet.

ADR-12Every reader declares a class, and non-promoting classes change nothing ADR-13Telemetry is reduced at the source and stored in Kafka and ClickHouse ADR-14Classification is a bounded batch over an analytical replica of the catalogue ADR-15Declarative policy first; a predictive classifier earns its place in shadow ADR-16The platform classifies; the storage layer does not tier itself ADR-17Objects under 1 MB are classified and moved only as cohorts ADR-18Blind means no demotion: one canary drives both the brake and the alarm

Placement policy 4

The arithmetic every demotion must clear, what stops oscillation, where guards are enforced, and how a policy reaches production.

ADR-19Every demotion clears a per-object break-even over the destination's minimum duration, recorded with its inputs ADR-20Hysteresis: 14-day dwell, a 3× promotion bar, and no promotion on a first read ADR-21Guards are enforced at commit against a per-tenant epoch, not at planning ADR-22Policy ships through shadow, dry run, rings and a hard movement cap, orchestrated by Temporal

Movement 3

How an object changes place without ever being lost, who does the work, and what a delete does to a move in progress.

ADR-23Copy, verify, commit, release, with the catalogue row as the state machine ADR-24Movers are a Go fleet on budget leases that name dollars and requests ADR-25A delete wins over a movement, and offboarding is proven against storage

Recall and cost 3

How cold data comes back, who pays for it, and how the saving is counted so that it survives a finance review.

ADR-26Recall is a Temporal job planned by pack and cartridge, and staging is not promotion ADR-27Retrieval budgets per tenant have a hard ceiling; legal authority is never refused and never free ADR-28Savings are net, against an all-hot counterfactual, attributed to decisions and reconciled

Security and operations 4

Who may move and who may delete, how holds change, how disagreement with storage is settled, and where it all runs.

ADR-29Movement and deletion are different principals with fifteen-minute, one-tier credentials ADR-30Restrictive hold and pin changes apply at once; relaxing changes need two approvers ADR-31Reconciliation believes the tier for where bytes are and the catalogue for where they should be ADR-32Two data centres and a witness, with cross-site failover as a rehearsed runbook

Technology by capability

Every capability on the views, the package that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns, with no managed service and no external dependency on any read path.

Open source This design
CapabilityChoiceOriginCredible alternativeWhy this oneRecord
Placement resolution Go resolver with a bounded in-process cache This design Direct catalogue queries from the file service One dependency (vtgate); staleness made safe by the release hold ADR-01
Stale-safe reads 24-hour release hold on every movement This design Primary-only reads; cache invalidation Lets caches and replicas serve resolution without a data-loss path ADR-03
Placement catalogue Vitess on MySQL 8.4, multicolumn vindex Open source TiDB; FoundationDB Tenant sharding, cheap point reads, online resharding, CNCF governance ADR-02
Disk tiers Ceph RGW, separate clusters per tier, multisite Open source MinIO; SeaweedFS S3, STS, SSE-KMS and notifications at exabyte scale, with open governance ADR-07
Tier ladder CUE in Git, signed with cosign Open source Hard-coded tier logic Prices and minimum durations as reviewed, versioned data ADR-06
Small-object storage Platform pack format, one tenant and key per pack This design tar; SeaweedFS volumes Per-entry identity and checksum; readable without the platform ADR-08
Archive tier EOS and CERN Tape Archive on LTO-10 Open source LTFS with own scheduler; dCache Retrieval queues, drive sharing and repack already solved at scale ADR-10
Access telemetry Apache Kafka with KRaft Open source Redpanda; NATS JetStream Replayable partitioned log shared with CDC and decisions ADR-13
Aggregates and analytical replica ClickHouse Open source Apache Druid; Trino on Iceberg Rollups at ingest and fast scans over billions of rows ADR-14
Catalogue change data Vitess VStream into Kafka Open source Nightly snapshot to Parquet Shard-aware and minutes behind, with no load on primaries ADR-14
Classification and break-even ClickHouse SQL and a Go evaluator This design Spark batch No extra cluster; the arithmetic lives in one auditable place ADR-19
Predictive classifier (Phase 2) LightGBM batch scoring in Python Open source XGBoost; survival models Tabular features, fast scoring, importances an auditor can read ADR-15
Freshness gate and dead-man Canary events, Prometheus, self-hosted Healthchecks Open source Alertmanager watchdog only Brake and alarm read one signal; the watcher lives outside the estate ADR-18
Policy rollout and recall jobs Temporal on PostgreSQL Open source Argo Workflows; job tables Durable timers, signals and multi-day workflows ADR-22
Movers and budget leases Go workers and a Go budget authority This design rclone; RGW sync modules Conditional commits and money-denominated limits need catalogue-aware code ADR-24
Release and cleanup Go release gate under a separate identity This design Deletes inside movers The only delete permission, conditioned on committed state ADR-23
Holds, pins, budgets PostgreSQL 17 with Patroni Open source Rows in Vitess Small relational data with approval workflows ADR-30
Decision and movement records Parquet on Ceph with S3 Object Lock Open source Kafka tiered storage Immutable, cheap, and readable by any engine for three years ADR-19
Workload identity and storage credentials SPIRE JWT-SVIDs exchanged at RGW STS Open source Static RGW keys rotated by OpenBao Attested, 15-minute, one-tier credentials ADR-29
Encryption keys OpenBao transit for RGW SSE-KMS Open source HashiCorp Vault Per-tenant keys under Linux Foundation governance ADR-29
Human identity and approvals Keycloak with WebAuthn Open source Dex; Authentik Existing OIDC provider with strong second factors ADR-30
Drift detection RGW bucket notifications and a rolling scan Open source Daily full scans Minutes for common changes, 14 days for everything ADR-31
Compute platform Kubernetes (RKE2) with Cilium, etcd witness Open source OKD; kubeadm Separate serving and tiering clusters with room-aware scheduling ADR-32

The decisions, and the alternatives that lost

Read path and catalogueWhat a read is allowed to depend on, where placement truth lives, and why stale answers are safe.

ADR-01

The read path never consults the tiering plane

Accepted

What is a read allowed to depend on?

Context
Every product read has to find its object, so whatever answers that question sets a ceiling on the product's availability and a floor on its latency. The tiering plane (telemetry, classifier, planner, movers) is large, batch-shaped and changes weekly. If a read ever calls it, even as a fallback, the storage bill and the product's availability become the same number.
Decision
Placement resolution is a lookup in the committed catalogue and nothing else. It does not evaluate policy, call the classifier, check a hold or probe a tier. The serving plane (resolver, read router, placement API, recall API, catalogue) is a separate Kubernetes cluster with a 99.99% target; the tiering plane is another cluster with a 99.5% target and no component the serving plane calls.
How it works on-premise
The resolver is a Go service whose only network dependency is vtgate. Its deployment manifests have no service entries for Kafka, ClickHouse, Temporal or the policy service, and a CI check fails any change that adds one. Access events are written to a local bounded buffer and shipped asynchronously; if Kafka is down the buffer drops events, never reads.
Options weighed
  • ChosenCatalogue lookup only, tiering plane fully separate: The tiering plane can be switched off for a week with no effect on reads. Its availability target can be honest.
  • RejectedResolve by probing tiers in order: Needs no catalogue, but puts up to five storage calls on a miss and makes an archive probe part of a hot read.
  • RejectedResolver consults policy for promote-on-read: Convenient, and it puts the policy engine on every read. Promotion is decided later from telemetry instead.
  • Right elsewhereProvider or storage-layer transparent tiering: Right when one storage system owns all tiers and its own metadata. Here tiers span Ceph clusters and tape.
Consequences
What it buys
  • Classifier, planner and mover outages are cost events, never availability events
  • The tiering plane can be redeployed, rewritten or paused without a change window for reads
  • The read path is small enough to reason about completely
What it costs
  • A cold read cannot promote itself synchronously; promotion waits for the next cycle
  • Every read emits telemetry through a buffer that must be sized and watched
  • The catalogue becomes the one component whose availability everything inherits
Choose differently when
A system where every tier is inside one storage product that already tracks class per object, and where the product's own metadata service is the read path anyway, gains nothing from a separate catalogue and should use the product's placement.
Why it holds up over time
The boundary is about dependency direction, not technology. Whatever replaces the classifier, the movers or even the storage tiers, the rule that reads depend only on committed placement stays correct and keeps the replacement low-risk.
LessonPut the system that decides on a different availability budget from the system that serves, and forbid the call between them rather than hoping nobody makes it.
Shown on views02 07 08 11
ADR-02

The placement catalogue runs on Vitess over MySQL, sharded by tenant

Accepted

Which open-source store can hold 9.2 billion placements, answer point reads at 1.4 million a second, and commit a conditional update with no data loss inside a data centre?

Context
The requirement asks for strong consistency, RPO 0 in a data centre, no cross-tenant transactions, 3.7 TB of rows before indexes, 34% growth a year, and a p99 of 8 ms. Every operation is either a point read by key or a single-row or single-tenant conditional write. There are no joins across tenants, no range scans on the hot path and no analytical queries, because those go to a replica (ADR-14).
Decision
Vitess on MySQL 8.4, 64 shards to start, keyed by a multicolumn vindex of tenant hash and object hash. Each shard has a primary and two semi-sync replicas in separate rooms of DC-A, and one asynchronous replica in DC-B. VTOrc handles in-DC failover. Commits are compare-and-set updates on row_version and guard_epoch within one shard.
How it works on-premise
MySQL runs on bare-metal NVMe servers, not in Kubernetes, managed by the Vitess operator's topology in etcd with the third etcd voter at the witness site. vtgate runs in the serving cluster. Resharding is done online with MoveTables and Reshard workflows as tenants grow. Backups use Vitess's built-in backup to a Ceph bucket that is not a placement tier.
Options weighed
  • ChosenVitess on MySQL: Tenant sharding is its native model; proven for this exact kind of product at Slack and GitHub; CNCF graduated; point reads are as cheap as MySQL gets.
  • Right elsewhereTiDB: Automatic region splitting and no shard key to design. Right for a team that will not own a sharding scheme; costs more per point read through its SQL and coprocessor layers.
  • Right elsewhereFoundationDB with a record layer: Strict serialisability across keys and a strong record at Apple and Snowflake for metadata. Needs a layer built and owned in-house and skills that are hard to hire.
  • RejectedCockroachDB: Technically suitable, but no longer released under an open-source licence, which rules it out for this stack.
  • RejectedCassandra or ScyllaDB: Excellent for point reads; conditional commits need lightweight transactions, which are slow and easy to misuse on the one write that must be exact.
Consequences
What it buys
  • Single-shard transactions are exactly the shape of every commit
  • Semi-sync across rooms gives RPO 0 in DC-A with plain MySQL operations knowledge
  • Online resharding lets the catalogue grow with the corpus without a migration project
What it costs
  • The shard key is a design decision the team owns forever
  • 256 MySQL instances is a real fleet to patch and watch
  • Cross-site failover is a runbook rather than automatic, by choice (ADR-32)
Choose differently when
Choose TiDB when the team cannot own a sharding scheme and point-read cost is not the constraint. Choose FoundationDB when cross-key transactions become necessary, for example if placements start spanning tenants.
Why it holds up over time
The catalogue's access pattern is key-value with conditional writes, so its schema moves to any store with that capability. MySQL's replication and storage engine have been stable for fifteen years, and Vitess's governance under the CNCF does not depend on one company's roadmap.
LessonPick the store for the operations you actually run on the hot path, and send everything else to a replica built for it.
Shown on views08 13 22
ADR-03

Bounded-stale resolution, made safe by a 24-hour release hold

Accepted

How can resolution use caches and replicas when the requirement says placement must be strongly consistent?

Context
1.4 million resolutions a second against primaries alone would need a much larger catalogue fleet, and the 4× burst would need more still. Caches and replicas are the obvious answer, and the obvious danger is a stale answer pointing at a location whose bytes were just deleted. The property that actually matters is not linearisable reads. It is that a resolved location always holds the bytes.
Decision
Sources are released no earlier than 24 hours after commit. Resolution may therefore use a 60-second in-process cache and replicas lagging less than 5 seconds, and falls back to the primary only when a row is not found. Commits remain serialisable on the primary. The design guarantees that every answer, however stale within 24 hours, names bytes that exist.
How it works on-premise
The movement row carries release_after = commit_time + 24 h. The release gate selects only rows past that time. vtgate routes resolver reads to replicas with a lag threshold of 5 seconds and to the primary on not-found. The resolver cache is a bounded LRU with a 60-second TTL and no negative caching.
Options weighed
  • ChosenStale reads, release held 24 h: Safety comes from ordering, not from read consistency. The hold also turns every movement into a reversible one for a day.
  • RejectedPrimary reads only: Linearisable and several times the catalogue hardware, and the burst still lands on primaries that also take commits.
  • RejectedCache with invalidation on commit: Invalidation across 48 pods is a distributed protocol with its own failure modes, protecting against a problem the hold removes.
  • RejectedRelease immediately after commit: Saves one day of duplicate storage and makes every cache, replica and long stream a data-loss path.
Consequences
What it buys
  • Most resolutions never reach a primary
  • Reversing a bad movement within a day is a pointer change with no retrieval
  • Long streams that started before a commit finish from the source
What it costs
  • One extra day of storage for every moved byte, about $250 to $600 a day at planned volumes
  • A newly committed placement may be read at its old location for up to 60 seconds
  • The guarantee depends on the release gate honouring release_after, so the gate is tested harder than anything else
Choose differently when
Where storage is so expensive that a day of duplicates matters, or where movements are rare enough that primaries can serve every read, drop the hold and read from primaries.
Why it holds up over time
This is a property of ordering: no cache, replica or storage technology can break it as long as release waits longer than the longest staleness. Future caches or read paths only need to state their staleness bound and check it is under 24 hours.
LessonWhen readers can be stale, make the writer wait out the staleness before destroying anything they might still point at.
Shown on views11 18 31
ADR-04

The catalogue decides, the tier is fact, and both can be rebuilt from storage

Accepted

Where does placement truth live: in the catalogue, in the storage layer's own metadata, or in both?

Context
A catalogue-only design is fast and loses everything if the catalogue is lost. A storage-only design is rebuildable and puts a listing over nine billion objects on the read path. The requirement wants catalogue reads on the hot path, a rebuild within 30 hours, and drift detected rather than absorbed.
Decision
The catalogue is authoritative for where to read. The storage tiers are believed about where bytes physically are. The two are reconciled continuously (ADR-31). Everything needed to rebuild location is written into storage itself: object keys encode tenant, object and version, and every pack carries its own index plus a sidecar manifest.
How it works on-premise
Unpacked keys follow t/<tenant>/<object>/<version> in per-shard buckets. Packs are written as p/<tenant>/<pack_id> with a footer index and a manifest object beside them. Tape files in CTA are named by pack id, and pack manifests stay on disk. A rebuild lists buckets, reads manifests in parallel and bulk-loads Vitess.
Options weighed
  • ChosenCatalogue authoritative, storage self-describing, reconciled: Fast reads, a bounded rebuild, and disagreements surface as findings.
  • RejectedCatalogue only, opaque storage keys: Smaller keys and nothing to scan, and a lost catalogue is a lost corpus.
  • RejectedStorage metadata only: Rebuildable by definition, and puts storage listing and class metadata on every read.
Consequences
What it buys
  • A full catalogue loss costs about 30 hours and the decision history, not objects
  • Anyone with storage access and the key convention can find any object without this software
  • Drift becomes measurable because both sides can be enumerated
What it costs
  • Key conventions are a permanent contract; changing them means rewriting objects
  • Pack manifests are extra objects to store and keep consistent
  • A rebuild restores location but not why an object is where it is
Choose differently when
When a storage product offers a transactional, queryable metadata service with its own replication, and every tier lives in it, the product's metadata can be the truth and the catalogue a cache.
Why it holds up over time
Self-describing data outlives every index built over it. In ten years the catalogue may be a different database, but objects named by their identity and packs that list their contents will still be readable by whatever replaces it.
LessonAn index should be something you can lose. Write enough into the data itself that the index can be rebuilt from it.
Shown on views12 30 31
ADR-05

Callers see a recall class and a job handle, never a tier

Accepted

What does a product know about where an object is, and what happens when bytes are not immediately available?

Context
If a product sees tier names it will build behaviour on them, and every ladder change becomes a product change. If a read of an archived object blocks until the bytes arrive, a four-minute or twelve-hour request sits on the read path. The requirement sets a 3-second synchronous ceiling and asks for synchronous-or-job to be decided.
Decision
Resolve returns one of three recall classes: immediate, delayed or scheduled. Immediate objects are read inline. For delayed and scheduled objects the caller receives, within 3 seconds, a job handle with an estimated availability time and a callback, never a hung request. Tier names, cluster names and vendor classes never appear in any product-facing response.
How it works on-premise
The recall class is a column of the ladder version, so the resolver maps tier to class from the placement row with no extra lookup. The recall API creates a Temporal workflow for the job and returns its id and the current per-tier estimate published by the quota planner.
Options weighed
  • ChosenRecall classes with job handles: Honest about physics, stable across ladder changes, and bounded on the read path.
  • RejectedSynchronous recall for every class: Simplest for callers, and puts a four-minute request, and on the deep rung a multi-day one, on the read path.
  • RejectedExpose tiers to product teams: Gives products more control and couples every product to the current storage estate.
Consequences
What it buys
  • Adding or retiring a tier changes no product code
  • No request is held open beyond 3 seconds
  • Products can design a 'fetching' state against three stable classes
What it costs
  • Every product that can read archived objects needs a fetching state and a callback
  • Estimates must be accurate enough to be believed, so the quota planner is part of the contract
  • A cold object that should have been immediate shows up as a product-visible delay, which is the point, and also a complaint
Choose differently when
If archived data is only ever read by batch tools, never by people, the job model can be the only model and immediate reads need no class at all.
Why it holds up over time
Recall classes describe what a caller experiences, which changes far less often than the media underneath. Tape, disk, a future glass or DNA archive: each maps to one of three classes, and the product never notices.
LessonExpose the service level a caller can plan around, not the mechanism that currently provides it.
Shown on views10 11 20

Tiers and storageWhat a tier is when the organisation owns the hardware, which software provides each rung, and how small objects are stored.

ADR-06

The tier ladder is signed, versioned configuration priced in internal transfer prices

Accepted

What does 'price per GB', 'retrieval fee' and 'minimum storage duration' mean when the organisation owns the disks and the tapes?

Context
The requirement defines placement arithmetic in cloud terms. On-premises there is no invoice per retrieval, but the costs are real: erasure-coded reads across twelve disks, drive-hours and robot mounts, dead space in immutable packs and on tape until reclamation. If those costs are not priced, the classifier optimises against a ladder that says reads and deletes are free, and it will be wrong in exactly the ways that cost most.
Decision
The ladder is an ordered list of tiers, each with storage price per GB-month, retrieval price per GB and per mount, minimum duration, first-byte latency, recall class and durability. Prices are internal transfer prices published by the storage team, fully loaded over five years. Minimum durations equal each medium's reclaim cycle. The ladder is versioned, signed, and every placement records the version it was made under.
How it works on-premise
The ladder is a CUE file in Git, reviewed by the storage team and finance, signed with cosign, and loaded by the classifier, the break-even evaluator and the cost ledger. A new version is evaluated in shadow before it applies to new decisions. Existing placements are never re-evaluated merely because a price changed.
Options weighed
  • ChosenTransfer-priced ladder as versioned configuration: The arithmetic stays identical to the cloud version, and the prices have a named owner.
  • RejectedTreat on-prem storage as sunk cost; tier by capacity pressure: Common, and it makes every read and delete look free until the next hardware order.
  • RejectedHard-code tier behaviour in the classifier: Faster to start, and every hardware change becomes a code change nobody reviews for cost.
Consequences
What it buys
  • Placement decisions and the cost ledger agree because they read the same numbers
  • Hardware refreshes and new media are configuration changes evaluated in shadow
  • Storage team, finance and platform argue about one table instead of three spreadsheets
What it costs
  • Transfer prices are estimates and will be disputed
  • Two ledgers (placement-time price and current price) must be kept to explain year-on-year changes
  • A wrong price produces systematically wrong decisions until corrected
Choose differently when
If the organisation later moves the corpus to a public cloud, the ladder is populated from the provider's price list instead, and nothing else in this record changes.
Why it holds up over time
Media prices halve every few years and tape generations change every two or three. A ladder that is data absorbs those changes without touching code, and the version stamp on every placement keeps old decisions explainable under the prices that justified them.
LessonIf a cost is real but invisible, price it explicitly, or your optimiser will spend it for you.
Shown on views09 17 26
ADR-07

Ceph RGW provides the disk tiers, one cluster set per tier, with lifecycle transitions disabled

Accepted

Which open-source object store provides hot, warm and cold, and are they one cluster with storage classes or separate clusters?

Context
The disk tiers hold about 32 PB logical across NVMe replicas, HDD erasure coding and dense HDD, in two data centres. They need an S3 API, short-lived credentials, server-side encryption with an external KMS, bucket notifications and asynchronous replication between sites. Ceph RGW can express tiers as storage classes inside one zone and move objects between them with its own lifecycle rules, which is exactly the out-of-band movement the requirement calls a governance failure.
Decision
Ceph RGW for all disk tiers, with each tier as its own set of Ceph clusters, replicated to DC-B with RGW multisite. Movement between tiers is always a copy made by the platform, never an RGW storage-class transition. Lifecycle transition rules are disabled on every placement bucket and detected if added.
How it works on-premise
Hot clusters use NVMe OSDs with 3× replication; warm uses HDD with EC 8+3; cold uses 20 TB HDD with EC 12+4 and larger bucket index shards for pack objects. Each cluster is sized in cells of 10 to 15 PB so a cluster rebuild or upgrade has a bounded blast radius. RGW STS, SSE-KMS with OpenBao and Kafka bucket notifications are enabled on all of them.
Options weighed
  • ChosenCeph RGW, separate clusters per tier: Mature at exabyte scale, open governance, S3 and STS compatible; tier failures and upgrades are isolated.
  • RejectedOne Ceph cluster, tiers as storage classes: Cheaper to run and allows in-place transitions, but one cluster failure takes every tier and transitions bypass the catalogue.
  • RejectedMinIO: Fast and simple, but its community edition was cut back in 2025 and its roadmap belongs to one company.
  • Right elsewhereSeaweedFS: Haystack-style volumes are excellent for small objects; right for a smaller estate that wants packing built in and accepts a smaller operator community.
Consequences
What it buys
  • A warm-tier incident never affects hot reads
  • Each tier upgrades on its own schedule and hardware generation
  • No storage-side feature can move an object the catalogue did not decide to move
What it costs
  • Every movement between tiers is a network copy rather than a metadata change
  • More clusters to run, monitor and upgrade
  • Ceph expertise is required at a depth the storage team must already have
Choose differently when
With a small corpus and a small storage team, one Ceph cluster with storage classes and a catalogue-driven transition API is simpler and the isolation is not worth the extra clusters.
Why it holds up over time
The platform uses Ceph only through the S3 API, STS and notifications. Any future S3-compatible store can replace one tier at a time by adding a ladder version, and the self-describing key convention (ADR-04) makes the migration a copy like any other movement.
LessonIf a storage system can move data on its own, turn that off when something else is keeping the record of where data is.
Shown on views09 22 30
ADR-08

Small objects are stored in single-tenant, single-key, self-describing packs

Accepted

Is the unit of tiering the object or the pack?

Context
62% of objects are below 128 KB and the median is 240 KB. On erasure-coded HDD and on tape, per-object metadata, allocation overhead and requests cost more than the bytes. Packing fixes that and creates new problems: a cold read becomes a ranged read into a large object, a delete becomes dead space, and a mixed-tenant pack turns offboarding, encryption and legal disclosure into compaction events.
Decision
The object remains the unit of placement and every object keeps its own catalogue row. Below warm, objects under 1 MB are stored in packs. A pack holds exactly one tenant's objects under one encryption key, grouped by folder and upload time, and seals at 2 GB or after 7 days open. The pack format is the platform's own: entry headers with identity, length, key id and CRC32C, a footer index, and a sidecar manifest.
How it works on-premise
The pack builder streams members from hot into a multipart upload on the cold cluster, reads the sealed pack back and checks every entry, writes the manifest, then commits member placements in batches of 1,000 rows per shard transaction. On tape, a pack is one CTA file. Reads use a ranged GET for one member's offset and length.
Options weighed
  • ChosenOwn pack format, one tenant and one key per pack: Offboarding, crypto-shredding and disclosure stay per tenant; the format is simple enough to read without this software.
  • RejectedMixed-tenant packs: Better fill for 180,000 small tenants, and a single tenant's deletion or key destruction becomes a compaction of everyone else.
  • Rejectedtar or zip archives: Familiar, but no per-entry checksum or key id in the header and no index designed for ranged reads.
  • Right elsewhereStore small objects individually: Right when the median object is megabytes and tiers charge nothing per object.
Consequences
What it buys
  • Millions of small-object requests become one pack write
  • A pack can be verified and read without the catalogue
  • Tenant-level deletion and key destruction never touch another tenant
What it costs
  • Small tenants produce small, underfilled packs
  • Deletes leave dead space that only compaction reclaims
  • A pack format is code the platform owns for as long as packs exist
Choose differently when
A single-tenant corpus, or one where encryption and deletion are not tenant-scoped, should pack across tenants for density.
Why it holds up over time
A format with a header per entry, a checksum and an index at the end is the design of tar, Haystack and every log-structured store before them. It will be readable in twenty years with a hundred lines of code, which is the standard an archive format should meet.
LessonPack along the boundaries along which you will later have to delete, encrypt and disclose.
Shown on views13 15
ADR-09

Compact a pack only when dead space passes half and minimum durations have expired

Accepted

When is it worth rewriting a pack to reclaim the space deleted members left behind?

Context
Compaction reads every live member of a pack and writes them into a new one. On cold disk that is an erasure-coded read of up to 2 GB; on tape it is a mount, a read and a rewrite. It is the most expensive operation the platform performs, and a naive threshold turns the platform's own housekeeping into its largest retrieval bill.
Decision
A pack is compacted only when dead bytes exceed 50% of its size and the minimum duration of its live members has passed. Packs are cut by cohort so that most expire together, and a pack whose live bytes reach zero is deleted without compaction. On tape, compaction is CTA's repack of whole cartridges, scheduled with generation migrations.
How it works on-premise
Dead bytes are tracked per pack as deletes commit. A daily job lists packs over the threshold, prices compaction with the ladder's retrieval price and the break-even evaluator, and submits only those where reclaimed storage over the next minimum duration exceeds the compaction cost.
Options weighed
  • ChosenThreshold of 50% dead plus break-even check: Compaction happens only when it pays, and cohort packing makes whole-pack expiry the common case.
  • RejectedLow threshold, around 20%: Keeps storage tidy and repeatedly re-reads live data to reclaim little.
  • RejectedNever compact: Zero retrieval cost and unbounded dead space for long-lived tenants with steady deletion.
Consequences
What it buys
  • The expensive operation runs only when it saves money
  • Most packs disappear whole at the end of their cohort's life
  • Tape compaction aligns with migrations that must happen anyway
What it costs
  • Up to half of a pack's bytes may be dead for a long time
  • Dead-space accounting must be exact to be trusted
  • Tenants with churn-heavy folders produce packs that are costly to keep
Choose differently when
If storage becomes more expensive than reads, as with very dense but slow media where retrieval is cheap, lower the threshold.
Why it holds up over time
The rule is an inequality over ladder prices, not a constant. When prices change, the threshold effectively moves with them without anyone editing it.
LessonHousekeeping that reads data is still retrieval. Price it like any other decision.
Shown on views15 13
ADR-10

The archive rung is LTO tape behind EOS and CTA, driven through the WLCG Tape REST API

Accepted

What provides a delayed-recall archive tier on-premises, with open-source software, at about 8 PB and growing?

Context
At archive prices only tape competes on-premises, and tape's hard problem is not writing but reading: queuing thousands of retrieval requests, ordering them by cartridge and position, sharing a handful of drives between interactive and bulk demand, and repacking cartridges across generations. Building that scheduler is where in-house tape projects fail.
Decision
The CERN Tape Archive manages the libraries, drives, queues and repack, with EOS as its disk buffer. The platform writes sealed packs into EOS, which archives them through CTA to two tape pools in two libraries in different data centres. Retrieval is requested through the WLCG Tape REST API. Drive reservations by demand class are CTA configuration derived from the ladder.
How it works on-premise
Two libraries with eight LTO-10 drives each, one per data centre. EOS provides about 500 TB of buffer on the cold Ceph hardware generation. The recall workflow submits stage requests per pack, polls for completion, and hands staged packs to unpack workers.
Options weighed
  • ChosenEOS and CTA with the Tape REST API: Built for exabyte-scale physics archives with the same retrieval problem; open source under CERN; the API is shared with dCache.
  • Right elsewhereLTFS with an in-house scheduler: Simple, portable cartridges; right for a small archive, and the scheduler becomes the project.
  • RejectedBareos or Bacula on tape: Backup software, designed for restoring jobs, not for serving recalls of individual packs.
  • RejectedDeep HDD or spun-down disk instead of tape: Faster recalls and roughly three to five times the cost per GB at this size.
Consequences
What it buys
  • Retrieval ordering, drive sharing and repack come from a system that already does them at scale
  • Two copies in two sites without a second product
  • A standard API means dCache could replace EOS and CTA
What it costs
  • CTA experience is rare outside research computing
  • EOS is another storage system for the storage team to run
  • Recall latency is minutes to hours and cannot be engineered away
Choose differently when
Below a few petabytes of archive, or without staff willing to run tape, deep HDD tiers in Ceph with spun-down pools are simpler and the cost difference is small.
Why it holds up over time
LTO has a published roadmap of roughly doubling capacity per generation, and CTA's repack moves data forward generation by generation. The platform only ever sees pack identifiers and the Tape REST API, so a change of tape software or media is invisible above the ladder.
LessonFor tape, buy or adopt the scheduler. Writing to tape is easy; reading back fairly under contention is the product.
Shown on views09 20 21 22
ADR-11

The ladder ends in a second custodian, in Phase 3, only for data kept as an obligation

Accepted

Does the ladder end inside the organisation's own estate, or with an independent custodian?

Context
Two tape copies in two data centres protect against media and site loss, but not against a failure that affects the organisation's whole estate: a ransomware operator with administrative access, a control-plane bug that deletes across both sites, or a commercial or legal dispute. A second custodian adds independence and costs an operational estate of its own, a courier contract and a recall time measured in days.
Decision
A deep-archive rung is added in Phase 3: a third tape copy of packs that are older than seven years or under regulatory retention, exported from CTA and held in an independent vault. It is scheduled recall with a 72-hour contract and is priced in the ladder. Data that is kept for business value rather than obligation does not go there.
How it works on-premise
A dedicated CTA tape pool writes the third copy; cartridges are ejected monthly and shipped under the vault contract with a manifest of pack ids. The catalogue records the vault location as a placement; the reconciler checks it against the monthly vault inventory rather than by scanning tapes.
Options weighed
  • ChosenVault custodian for obligation data, Phase 3: Independence where it is legally or existentially needed, not for the whole corpus.
  • Right elsewhereSecond cloud as deep archive: Faster recall and no courier; right if the organisation already has a cloud agreement and accepts data leaving its estate.
  • RejectedEnd the ladder at on-site tape: Cheapest, and leaves the whole archive exposed to one administrative domain.
Consequences
What it buys
  • A copy exists that no administrator of this estate can delete
  • The rung is priced and visible, so it is used deliberately
  • Phase 3 timing lets tape operations mature first
What it costs
  • Recall in days, and a contract to manage
  • A monthly physical process that must be audited
  • A third copy of obligation data adds cost that has no saving attached
Choose differently when
If the oldest data is a business asset read occasionally rather than a compliance obligation, a second cloud with hours-long recall is the better custodian.
Why it holds up over time
Separation of administrative domains is the one protection that does not weaken as software changes. Whatever runs this platform in ten years, a cartridge in a vault it cannot reach is still out of reach.
LessonReplicas protect against failure; custodians protect against the people and systems that manage the replicas.
Shown on views09 22 27

Signal and classificationWhat counts as evidence of use, how it is collected cheaply, who classifies, and what happens when the signal goes quiet.

ADR-12

Every reader declares a class, and non-promoting classes change nothing

Accepted

How does the platform stop a training job, a migration or an eDiscovery sweep from making the whole corpus look hot?

Context
An access log cannot tell a user opening a file from a pipeline reading every file. A single corpus-wide scan resets last-access for every object, and a classifier that believes it undoes months of demotions and then pays to redo them. The only party that knows why a read happened is the reader.
Decision
Resolve requires a reader class: interactive, sync-client, preview-render, batch-export, ml-training, ediscovery or migration. The class is bound to the caller's workload identity, so a pipeline cannot claim to be interactive. Batch-export, ml-training, ediscovery and migration are non-promoting: their reads are recorded and costed but never update the promoting aggregate that classification reads.
How it works on-premise
Allowed classes per SPIFFE identity are policy data; the resolver rejects a class the caller's identity may not use. Access events carry the class; ClickHouse materialised views maintain promoting and non-promoting aggregates separately.
Options weighed
  • ChosenDeclared class bound to identity: The reader states intent, and identity keeps the statement honest.
  • RejectedInfer bulk reads from access patterns: Needs no caller change, and misclassifies bursts of genuine use and slow scans alike.
  • RejectedSeparate read endpoints for bulk callers: Clear, and multiplies interfaces while still trusting the caller to use the right one.
Consequences
What it buys
  • Scans and sweeps leave placement untouched
  • Retrieval costs can be attributed to the kind of reader that caused them
  • Tape access for training becomes an explicit authorisation rather than a side effect
What it costs
  • Every calling team must change its client to declare a class
  • Misconfigured identities surface as rejected reads during rollout
  • Class definitions become a governed list that will be argued over
Choose differently when
A corpus read only by people through one application can infer class from the application and skip the declaration.
Why it holds up over time
New kinds of bulk reader will keep appearing, AI training being only the latest. Each becomes a new class in policy data rather than a new source of false heat.
LessonWhen the meaning of an event depends on intent, make the caller declare the intent and bind the declaration to identity.
Shown on views03 14 21
ADR-13

Telemetry is reduced at the source and stored in Kafka and ClickHouse

Accepted

How does the platform collect access evidence for 1.4 million reads a second without telemetry becoming its largest cost?

Context
The requirement names telemetry as the platform's dominant controllable cost. A raw event per read at peak is far more than the 120 billion events a month the requirement budgets. Classification needs, per object or cohort and per class, the last access and the number of distinct access days in the window. It does not need every range read.
Decision
The resolver emits the first access per object, reader class and hour, with a count, and aggregates objects under 1 MB per folder cohort. Events go to Kafka and are rolled up by ClickHouse into raw events kept 90 days and aggregates kept three years.
How it works on-premise
A bounded in-process set per resolver pod, flushed every 60 seconds, provides deduplication. Kafka runs with KRaft on six brokers across rooms with replication factor 3. ClickHouse consumes through a Kafka engine table into ReplacingMergeTree for raw events and AggregatingMergeTree for aggregates, on a 12-node cluster.
Options weighed
  • ChosenDeduplicate at source, Kafka to ClickHouse: Cuts event volume by an estimated 15 to 25 times without losing any signal classification uses.
  • RejectedLog every read, aggregate downstream: Maximum fidelity for a question nobody asks, at the platform's largest cost.
  • RejectedKafka with Flink or Spark streaming: Powerful, and one more distributed system for a rollup ClickHouse does at ingest.
  • Right elsewhereSampled events only: Right for classes where last access does not matter; wrong where one read in 400 days is the whole signal.
Consequences
What it buys
  • Telemetry cost grows with distinct object-hours, not with read rate
  • Classification reads compact aggregates instead of scanning events
  • Ingest and aggregation scale independently
What it costs
  • Per-read detail is gone, so read-count questions have hourly resolution
  • A resolver pod crash loses up to 60 seconds of its events
  • ClickHouse becomes a system the platform team must operate well
Choose differently when
If per-read forensics are a requirement, for example for security investigations, send full events to a separate cheap log store and keep this pipeline for classification.
Why it holds up over time
The signal is defined as last access and distinct days per class, which does not depend on the transport. Kafka and ClickHouse can be replaced by any log and any columnar store; the aggregate definitions are the lasting asset.
LessonCollect the signal the decision uses, at the resolution the decision uses, as close to the source as possible.
Shown on views12 14
ADR-14

Classification is a bounded batch over an analytical replica of the catalogue

Accepted

Where does a classification cycle over 9.2 billion placements run, without degrading the store every read depends on?

Context
Classification joins every placement with its access aggregate, its holds and its dwell and minimum-duration dates. That is a full scan with joins. Run against the catalogue, it competes with 1.4 million reads a second. The requirement asks for the hot path and the analytical path to be physically separable, and for a cycle under 6 hours whose cost per object is published.
Decision
Catalogue changes stream through Vitess VStream into Kafka and into a ClickHouse replica of placement, movement and guard tables. Each cycle runs as SQL over the replica and the aggregates to produce candidates, and a Go evaluator applies break-even and guards. The cycle's compute cost and duration are published per run.
How it works on-premise
A Debezium-compatible VStream connector publishes row changes to Kafka topics per keyspace. ClickHouse applies them into ReplacingMergeTree tables keyed by object. The classifier job is a Kubernetes CronJob in the tiering cluster that writes decisions to Kafka and a plan to Temporal.
Options weighed
  • ChosenCDC replica in ClickHouse: Scans run on a store built for them, minutes behind the catalogue, with no load on it.
  • RejectedScan Vitess read replicas: No new system, and full scans on replicas that also serve resolution.
  • Right elsewhereNightly snapshot export to Parquet: Simpler, and a day stale. Right if cycles run daily and holds are always re-checked at commit, which they are here anyway.
Consequences
What it buys
  • No classification query ever touches a store on the read path
  • The replica also serves the cost ledger, reconciliation and ad hoc analysis
  • Cycle cost is a measured number that can be traded against frequency
What it costs
  • The replica lags the catalogue, so plans can be stale
  • A CDC pipeline to operate and keep in sync
  • A second copy of 3.7 TB of rows
Choose differently when
A corpus small enough to scan in minutes on one replica does not need a separate analytical store.
Why it holds up over time
Separating transactional from analytical access is one of the oldest stable patterns in data systems, and change data capture is how it is done without coupling the two. Staleness is made harmless by re-checking guards at commit (ADR-21).
LessonLet the analytical copy propose. Only the transactional store may permit.
Shown on views08 16
ADR-15

Declarative policy first; a predictive classifier earns its place in shadow

Accepted

Is placement decided by age-and-access rules or by a trained model?

Context
Rules are auditable, cheap and wrong at the margins in exactly the ways that cost retrieval fees. A model is better at the margins and cannot be explained to an auditor in a sentence. Regulated tenants may refuse placements nobody can justify per object, and nobody yet knows how much a model would actually save on this corpus.
Decision
The MVP uses declarative policies: per object class, an observation window, a minimum idle time, and the break-even inequality evaluated per object or cohort. A predictive classifier is introduced in Phase 2 in shadow only, scored against the declarative policy on realised break-even. It takes over for a class only after beating the rules for two consecutive 90-day scoring periods, and tenants may pin themselves to declarative policy.
How it works on-premise
Declarative policies are CUE evaluated by the classifier's SQL and Go evaluator. The Phase 2 model is a gradient-boosted classifier trained in Python on aggregate features exported to Parquet, scored in batch, with its outputs written as shadow decisions carrying the model version.
Options weighed
  • ChosenRules first, model in shadow, promoted on realised savings: Auditable from day one, and a model has to prove itself in money before it moves anything.
  • RejectedModel from the start: Possibly better at launch, with no baseline to prove it and no explanation for regulated tenants.
  • Right elsewhereRules forever: Right if the shadow comparison shows the model's marginal saving is smaller than its operating cost.
Consequences
What it buys
  • Every placement in the first year can be explained in one sentence
  • The model's value is measured rather than assumed
  • Regulated tenants keep auditable placement permanently
What it costs
  • Early margins are left on the table
  • Two classifiers run side by side during Phase 2
  • Promotion criteria must be defended when the model looks good but has not yet earned it
Choose differently when
If a similar corpus already has a proven model and scored history, start the model in shadow on day one and shorten the scoring period.
Why it holds up over time
Models will be retrained and replaced many times. The comparison framework (the same decisions, the same ladder, scored on realised money) is what persists, and it makes each replacement a measured decision.
LessonShip the explainable baseline first, and let the clever version earn production by beating it in money.
Shown on views16 23 27
ADR-16

The platform classifies; the storage layer does not tier itself

Accepted

Should the platform build its own classifier, or let the storage system's own tiering decide?

Context
The requirement's build-versus-buy question is Autoclass. On-premises its equivalents are RGW lifecycle transitions and commercial hierarchical storage managers. They need no telemetry pipeline. They also cannot see tenant, workspace, reader class or holds; they cannot be scored or overridden per tenant; and they move objects without the catalogue knowing.
Decision
The platform classifies. Storage-side lifecycle transitions are disabled on placement buckets (ADR-07). The decision is revisited in Phase 3 for a narrowly defined long tail if the per-object cost of the platform's own classification ever exceeds the saving it adds over an age-only rule for that tail.
How it works on-premise
RGW bucket policies forbid lifecycle transition configuration on placement buckets, and the reconciler treats a transition notification as a governance finding. A commercial HSM is not deployed.
Options weighed
  • ChosenOwn classifier with reader class and holds: Sees what storage cannot, is scored per decision, and honours holds and pins by construction.
  • RejectedRGW lifecycle transitions by age: Free to run, blind to reader class and holds, and moves objects the catalogue does not know about.
  • RejectedCommercial HSM (IBM Storage Scale, Quantum StorNext): Mature and proprietary, which the stack excludes; still blind to tenant context.
  • DeferredStorage tiering for an untenanted long tail: Reconsidered in Phase 3 if measured overhead says so.
Consequences
What it buys
  • Holds, pins and reader classes are honoured in every decision
  • Every movement is attributable and reversible
  • The classifier can be scored against a declared baseline
What it costs
  • A telemetry pipeline and classifier to build and run
  • The platform's own overhead must stay below what it adds
  • No vendor to call when classification is wrong
Choose differently when
A corpus with no holds, no tenant-level pins and no bulk readers, where an age-only rule is nearly optimal, should let the storage layer tier itself.
Why it holds up over time
Context the storage layer cannot see (who reads, why, and under what legal constraint) is permanent to this kind of product. As long as that remains true, the platform's classifier has an edge no storage feature can copy.
LessonBuy the classifier when the information it needs lives inside the product you buy. Build it when the information lives in your application.
Shown on views16 30
ADR-17

Objects under 1 MB are classified and moved only as cohorts

Accepted

What is the smallest object worth tiering on its own?

Context
The per-object overhead target is $0.0006 a year, covering catalogue share, telemetry, decisions and movement requests. Moving 1 GB from hot to cold saves about $0.19 a year. Below roughly 3 MB at the target overhead, or about 1 MB at the platform's estimated real overhead, an object costs more to tier individually than it saves. Objects under 1 MB are an estimated 75% of objects and 4% of bytes.
Decision
The cohort threshold is 1 MB, derived from the published per-object overhead and revised when that figure changes. Objects below it are aggregated per folder in telemetry, classified per folder, priced as the pack they would form, and moved only as pack members. In the MVP, which has no packs, they stay where they are.
How it works on-premise
The threshold is a ladder-adjacent policy constant read by the resolver's event emitter, the classifier and the pack builder. Cohort keys are tenant, folder and upload month. A cohort decision writes one decision record naming its members' count and byte total, and each member's placement row carries that decision id.
Options weighed
  • ChosenCohorts below 1 MB, individual above: Spends per-object machinery only where each object can repay it.
  • RejectedClassify every object individually: Most precise, and most of the decisions lose money before they are made.
  • DeferredNever tier small objects: The MVP position. Leaves about 7% of the target saving until packs exist.
Consequences
What it buys
  • Telemetry, decisions and requests scale with folders and bytes, not with object count
  • Movement budgets are spent where the money is
  • The threshold is a published number that can be argued with
What it costs
  • One hot object in a cold folder can hold back its cohort
  • Folder structure becomes an input to placement quality
  • About 7% of the saving waits for Phase 2
Choose differently when
When per-object overhead falls, or storage prices rise sharply, the threshold drops; at an overhead ten times smaller, individual classification of 100 KB objects pays.
Why it holds up over time
The threshold is computed from two published figures, overhead and tier price spread, so it moves as either changes. The rule 'do not spend more deciding than the decision can save' does not age.
LessonPublish your per-item overhead. It tells you which items are too small to be worth deciding about.
Shown on views16 17 27
ADR-18

Blind means no demotion: one canary drives both the brake and the alarm

Accepted

How does the platform know its access signal has stopped, and what does it do then?

Context
A stalled telemetry pipeline and a quiet corpus look identical: no events. A classifier running on stale aggregates sees everything as idle and demotes it, which the requirement names as the single most expensive failure mode. Monitoring that watches only for errors cannot see a pipeline that has stopped without failing.
Decision
A synthetic canary event is written into every Kafka partition every 60 seconds and aggregated like real events. The classifier's freshness gate refuses to produce demotion decisions when the newest canary visible in ClickHouse aggregates is older than 15 minutes. The same age is checked from the witness site by a self-hosted dead-man monitor that pages when it stops receiving pings. Promotion, recall and reads continue regardless.
How it works on-premise
A small Go job per partition emits canaries. A ClickHouse query exposes canary age as a Prometheus metric; the gate reads that metric, and a sidecar pings Healthchecks at the witness site only while age is under threshold. Demotion resumes after two consecutive fresh cycles.
Options weighed
  • ChosenOne canary, shared by gate and external monitor: The brake and the alarm cannot disagree about whether the pipeline is alive.
  • RejectedAlert on consumer lag and errors: Catches slow pipelines, and misses a producer that silently stopped sending.
  • RejectedPause demotion when event volume drops: Confuses holidays with outages and outages with quiet weeks.
Consequences
What it buys
  • A blind classifier cannot demote
  • The watcher does not depend on the estate it watches
  • Freshness is one number with one definition
What it costs
  • A canary pipeline to maintain and exclude from every business metric
  • Demotion pauses during planned maintenance too
  • Two clean cycles to resume adds up to 12 hours of lost demotion after an incident
Choose differently when
If demotion decisions were cheap to reverse, for example with no minimum durations and no retrieval charges, a stale classifier would be a nuisance and a simple lag alert would do.
Why it holds up over time
Dead-man signals are independent of the pipeline's technology. Replacing Kafka or ClickHouse means re-emitting the same canary, and the rule that absence of evidence halts irreversible action stays right.
LessonFor any automation that acts on the absence of events, prove the events can still arrive before letting it act.
Shown on views14 25 31

Placement policyThe arithmetic every demotion must clear, what stops oscillation, where guards are enforced, and how a policy reaches production.

ADR-19

Every demotion clears a per-object break-even over the destination's minimum duration, recorded with its inputs

Accepted

What exactly must be true for a demotion to be allowed, and how is it checked later?

Context
Age rules demote objects that are about to be read or deleted. The requirement's figures imply heavy churn (210 TB a day ingested against 38 TB a day of net growth), so for young objects early deletion is often a larger risk than retrieval. A decision without recorded inputs cannot be audited, scored or reversed by population.
Decision
A demotion is allowed only if the storage saving over the destination's minimum duration exceeds expected retrieval cost at the predicted read probability, plus movement cost, plus expected early-deletion charge. The evaluation is per object or per cohort. Every decision records the policy version, the ladder version, the inputs, the predicted read probability, the break-even date, a confidence and the verdict, including refusals.
How it works on-premise
The Go evaluator reads candidate rows and ladder prices and writes decisions to a Kafka topic with acks from all in-sync replicas. ClickHouse keeps them queryable; hourly Parquet files with S3 Object Lock on a non-placement Ceph bucket keep them for three years, independently of the catalogue.
Options weighed
  • ChosenRecorded per-object inequality over minimum duration: Every bet has stated odds, and early deletion is priced rather than discovered.
  • RejectedAge threshold only: Simple and auditable, and systematically wrong for periodic and short-lived classes.
  • RejectedAggregate break-even per policy: Right on average, and hides the populations where the policy loses money.
Consequences
What it buys
  • Every retrieval and early-delete charge can be traced to a decision
  • A policy version's moved population is one query
  • Classifiers are scored on predicted against realised break-even
What it costs
  • About 1 to 2 billion decision records a year to store and query
  • Read and delete probabilities must be estimated, and bad estimates are recorded as bad
  • Refusals add volume that has no direct saving
Choose differently when
Where there are no minimum durations and retrieval is free, the horizon disappears and a simple idle threshold is nearly as good.
Why it holds up over time
The inequality is independent of tiers, media and prices, all of which come from the ladder. Records carrying their inputs let a future team re-price old decisions under new assumptions.
LessonIf a decision can cost money later, write down the odds it was made on at the time it was made.
Shown on views06 17 24
ADR-20

Hysteresis: 14-day dwell, a 3× promotion bar, and no promotion on a first read

Accepted

Does a cold read promote the object, and what stops objects oscillating between tiers?

Context
Promoting on every cold read pays a retrieval fee and then throws away the saving that justified the demotion, often with an early-deletion charge on top. Never promoting means a genuinely re-warmed object pays retrieval forever. Documents reopened once for a meeting are the common case: bursty, then quiet.
Decision
Every object stays at least 14 days in a tier before any further movement. Promotion requires expected retrieval savings over the source tier's minimum duration to exceed three times the demotion margin. A single read never promotes on its own; the default trigger is reads by a promoting class on at least two distinct days within 30 days, evaluated at the next cycle. Rehydration for a recall is not promotion.
How it works on-premise
dwell_until is a placement column set at commit and checked in the commit transaction. Promotion candidates come from the same classification cycle, flagged by the aggregates' distinct-day counts. Tenants may configure the trigger per object class within bounds set by policy.
Options weighed
  • ChosenDwell, 3× bar, second distinct-day read: Absorbs one-off reopenings and still catches objects that have genuinely warmed.
  • RejectedPromote on first cold read: Best latency for repeat reads, and pays retrieval plus early deletion for every one-off.
  • RejectedNever promote: Cheapest in movements, and a warmed corpus pays retrieval indefinitely.
Consequences
What it buys
  • No object can flip tiers faster than every 14 days
  • One-off reopenings cost one retrieval, not a round trip
  • Promotion thresholds are data, measured against second-read probability
What it costs
  • The second and third reads of a re-warmed object are still slow
  • A 3× bar will sometimes leave an object cold that should be hot
  • Users may notice a document that stays slow after they open it twice in one day
Choose differently when
If measured second-read probability after a first cold read is high, above about 40% within a week, promote on first read for that class.
Why it holds up over time
Hysteresis is how every control system avoids oscillation, and the bar and dwell are parameters that move with measured behaviour rather than constants that age.
LessonMake the move back harder than the move away, or the system will spend its savings changing its mind.
Shown on views24 20
ADR-21

Guards are enforced at commit against a per-tenant epoch, not at planning

Accepted

Where is 'no demotion below a hold's floor' actually enforced, given that plans are made from a replica minutes old and movements take minutes more?

Context
A hold can arrive at any moment between classification and commit. Checking holds only when planning leaves a window in which a movement planned before the hold commits after it. Writing a hold flag onto every affected placement row means updating up to hundreds of millions of rows for one custodian, which is slow exactly when speed matters.
Decision
Holds and pins are scope rows (workspace, folder, owner) stored beside the tenant's placements in the same Vitess shard, with a tenant_guard row holding guard_epoch. Adding or changing any scope increments the epoch. A movement records the epoch when it starts, and its commit is a single-shard transaction that fails if the epoch has changed or if the object falls under a scope forbidding the destination.
How it works on-premise
The commit is one transaction: read tenant_guard for update, compare the epoch, check matching scopes by the object's workspace, folder and owner columns, then update placement and movement. The policy service writes scope changes through the same shard routing.
Options weighed
  • ChosenPer-tenant epoch checked in the commit: No window between hold and enforcement, and one row written per hold.
  • RejectedCheck holds at planning only: Cheapest commit, and a hold placed during a movement is ignored.
  • RejectedHold flag on every placement row: Per-object precision, and a large hold becomes a long bulk update.
Consequences
What it buys
  • A hold is effective against the next commit, typically within a second
  • The mover needs no knowledge of holds
  • Guards cannot drift from the replica used to plan
What it costs
  • Any scope change aborts all of the tenant's in-flight movements, not only the affected ones
  • Scope matching depends on workspace, folder and owner columns being correct in the catalogue
  • Very active tenants with frequent pin changes see more wasted copies
Choose differently when
If a single tenant's holds change many times an hour, epochs per workspace rather than per tenant reduce wasted work at the cost of more guard rows to read.
Why it holds up over time
Optimistic concurrency with a version number is independent of database, language and storage. The rule that constraints are enforced where the state changes, not where the change was planned, holds for any future planner.
LessonEnforce a prohibition in the same transaction that would violate it.
Shown on views13 16 19
ADR-22

Policy ships through shadow, dry run, rings and a hard movement cap, orchestrated by Temporal

Accepted

How does a policy change that could move a hundred million objects reach production without becoming the incident?

Context
A tool that can relocate the corpus is a larger risk than the storage bill it exists to reduce. A dry run on a sample shows arithmetic, not behaviour: request cost, throughput and the populations a policy mis-models only appear when real objects move. A rollout also has to survive days of waiting between rings, restarts and human approvals.
Decision
Every policy version runs in shadow over the whole corpus, then a dry run copies a 1% stratified sample to scratch space without committing, then rings of 0.1%, 1%, 10% and 100% of eligible objects selected by stable hash. Each ring waits at least 72 hours and passes gates on re-read rate, wrong-tier rate, charge ratio and the gap to shadow projection. No policy moves more than 2% of objects or 1.5 PB in any 7-day window. One freeze flag stops all rings.
How it works on-premise
The rollout is a Temporal workflow per policy version with durable timers for ring waits, signals for approval and freeze, and activities that read gate metrics from ClickHouse. Plans are written to Kafka as batches for movers. Temporal runs on PostgreSQL in the tiering cluster.
Options weighed
  • ChosenShadow, dry run, rings, cap, in Temporal: Each step answers a different question, and the workflow survives the days between them.
  • RejectedDry run then full apply: Fast, and the first real evidence of a bad policy is the whole corpus moving.
  • Right elsewhereArgo Workflows or cron with state tables: Workable, with approvals, long waits and freezes hand-built around them.
Consequences
What it buys
  • The blast radius of a bad policy is bounded before anyone looks
  • Each ring produces measured evidence for the next
  • A single signal halts everything
What it costs
  • A full rollout takes at least two weeks
  • Temporal is a system to run and upgrade
  • The cap can delay a legitimately urgent policy, which must then be argued for explicitly
Choose differently when
For policies that only refuse or only promote, which cannot lose objects or incur early deletion, shadow and a single ring are enough.
Why it holds up over time
Progressive delivery with automated gates is how software, configuration and infrastructure changes are shipped safely, and it applies unchanged to whatever classifier or ladder comes next.
LessonTreat a policy that moves data like a deployment, including the cap on how much it may change at once.
Shown on views04 06 23

MovementHow an object changes place without ever being lost, who does the work, and what a delete does to a move in progress.

ADR-23

Copy, verify, commit, release, with the catalogue row as the state machine

Accepted

How does an object change place so that no failure at any instant leaves its only copy somewhere the catalogue does not name?

Context
A movement touches two storage locations, a database and possibly a second site, any of which can fail between steps. Workers die mid-copy. Commit acknowledgements get lost. Replication to the second data centre is asynchronous. The requirement sets object data RPO to zero and accepts paying twice to guarantee it.
Decision
Every movement is: mark in-flight in the catalogue with a movement id; copy to the destination with a checksum; verify; commit the new placement conditionally; and, no earlier than 24 hours later and only after confirming the destination exists in DC-B, release the source. The movement row records each state. A partial destination is garbage, never a state to repair. All steps are idempotent by movement id.
How it works on-premise
Movers write to the final destination key with the S3 CRC32C checksum header. Pack movements read back the whole pack. The release gate issues a HEAD against the DC-B zone before deleting the source in DC-A. A sweeper resumes movements stuck in copying for more than one hour, and abandoned ones are cleaned by the gate.
Options weighed
  • ChosenFour-step protocol with delayed, replica-checked release: At every instant the catalogue names at least one complete, verified copy in two sites.
  • RejectedCopy, commit and release in one step: Half the duplicate storage, and a crash between commit and release, or a lagging replica, can lose data.
  • RejectedTwo-phase commit across storage and catalogue: Storage does not participate in transactions, so it cannot be made atomic anyway.
Consequences
What it buys
  • No sequence of crashes can lose an object
  • Any worker can resume any movement from its row
  • Reads never notice a movement in progress
What it costs
  • Every moved byte is stored twice for at least a day
  • Orphaned copies are expected and must be found and cleaned
  • The release gate is a correctness-critical service
Choose differently when
Nothing about the ordering should flip. The 24-hour hold can shorten where caches and replicas are not used on the read path.
Why it holds up over time
The protocol needs only conditional writes, object PUT and DELETE, and a way to check a replica. Those exist in every storage and database generation, so the safety argument outlives every product in this design.
LessonNever destroy the old copy until the record says the new one exists, and the new one really does, in every place it is supposed to.
Shown on views18 22 31
ADR-24

Movers are a Go fleet on budget leases that name dollars and requests

Accepted

Who does the copying, and what stops movement from overrunning its budget or starving user traffic?

Context
900 TB and 120 million objects a day, 4× for a backfill, across Ceph clusters that also serve user reads and writes. At a 240 KB median, requests run out before bytes do. The requirement asks for a budget in currency and in requests per second, and for movement to yield to foreground traffic.
Decision
Movers are stateless Go workers consuming plan batches from Kafka. Before each batch a worker takes a five-minute lease from the budget authority naming bytes, requests and estimated cost against the daily budget and per-cluster request ceilings. Leases shrink automatically when Ceph foreground latency or queue depth crosses a threshold. Backfills run in a separate, lower-priority lease pool.
How it works on-premise
The budget authority is a small Go service with state in PostgreSQL and per-cluster token buckets fed by Prometheus metrics from RGW. Movers run on 24 dedicated nodes with 2 × 25 GbE in the tiering cluster, next to the storage network.
Options weighed
  • ChosenCustom Go movers with leased budgets: Catalogue-aware commits and money-denominated limits need code that understands both.
  • Rejectedrclone or RGW sync modules: Excellent copiers with no notion of conditional commits, holds or budgets.
  • RejectedSpark jobs for movement: Scales copies well and turns per-object idempotent state into job-level retries.
Consequences
What it buys
  • Movement spend is bounded per day in dollars and per second in requests
  • User traffic takes priority automatically
  • Workers are replaceable; all state is in the catalogue and Kafka
What it costs
  • Mover code is correctness-critical and owned in-house
  • Lease tuning interacts with storage performance and needs measurement
  • A backfill can take much longer than planned when foreground load is high
Choose differently when
A corpus with few, large objects and generous request limits can use an off-the-shelf copier and a thin commit wrapper.
Why it holds up over time
Leasing a budget before spending it is independent of media and prices, which come from the ladder. The fleet can move to new hardware or a new language without changing the contract it keeps with the catalogue.
LessonRate-limit automation in the unit that actually runs out, and let the thing it competes with take capacity back.
Shown on views08 18 21
ADR-25

A delete wins over a movement, and offboarding is proven against storage

Accepted

What happens when an object is deleted mid-move, and how does the platform prove a departing tenant's data is gone?

Context
A movement that commits after a delete resurrects the object. A delete that removes only the catalogue row leaves a destination copy nobody knows about. Tenant offboarding cancels thousands of in-flight movements at once, incurs minimum-duration charges, and needs proof of completeness that a catalogue alone cannot give.
Decision
Deletes write a tombstone into the placement row. Commits against tombstoned rows fail, and the release gate deletes both the destination copy and, after the release hold, the source. Tombstones are kept 35 days. Offboarding sets the tenant's guard to frozen, lets in-flight movements fail, accepts and attributes minimum-duration charges, and is complete only when a tier scan finds no bytes under the tenant's prefixes.
How it works on-premise
Tombstones are written by the placement API under the delete identity. The release gate processes tombstones and abandoned movements from catalogue queries per shard. Offboarding runs as a Temporal workflow ending with a scan report signed and stored in the records archive.
Options weighed
  • ChosenTombstone plus scan-proven offboarding: No resurrection, no forgotten copies, and completeness proven where the bytes are.
  • RejectedDelete row and bytes immediately: Simple, and a concurrent commit can recreate the row pointing at a copy.
  • RejectedProve offboarding from the catalogue: Fast, and blind to exactly the orphans a tenant's lawyers would ask about.
Consequences
What it buys
  • Deletes are final even under concurrent movement
  • Offboarding produces evidence, not an assertion
  • Orphan cleanup and delete cleanup share one path
What it costs
  • 35 days of tombstones in the catalogue
  • Offboarding waits for a scan
  • Minimum-duration charges on offboarding are accepted rather than avoided
Choose differently when
If deletion is legally required within hours rather than days, crypto-shred the tenant's keys first and treat the scan as confirmation.
Why it holds up over time
Tombstones and verification by enumeration are how distributed stores have handled deletion for decades. The completeness proof depends only on the self-describing key convention (ADR-04).
LessonProve that data is gone by looking where data lives, not where the index says it lived.
Shown on views19 31

Recall and costHow cold data comes back, who pays for it, and how the saving is counted so that it survives a finance review.

ADR-26

Recall is a Temporal job planned by pack and cartridge, and staging is not promotion

Accepted

How do objects that cannot be read immediately come back, individually and fifty million at a time?

Context
Tape recall takes minutes to hours, requests arrive individually and in huge batches, retrieval must be ordered by cartridge and position, budgets can run out mid-job, and bytes already retrieved have already been paid for. The requirement also insists that rehydrated bytes being temporarily available is not a placement change.
Decision
Every non-immediate read or bulk request is a recall job in Temporal. The job resolves placements, groups members by pack and cartridge, prices the job, obtains authority, submits stage requests, unpacks members into a staging bucket with a TTL (48 hours by default, up to 7 days) and reports unread bytes at expiry. Placement rows are not changed; a separate promotion decision may follow.
How it works on-premise
Temporal workflows with child workflows per 10,000 packs. The quota planner holds per-tier drive reservations and publishes estimated recall time per class every minute. The staging bucket is on the hot Ceph cluster and uses RGW lifecycle expiration, which is allowed here because staging is not placement.
Options weighed
  • ChosenTemporal jobs, pack and cartridge planning, TTL staging: Durable across restarts and pauses, efficient on tape, and clean about what is placement and what is a copy.
  • RejectedSynchronous recall: Simple for callers and puts hours on the read path.
  • RejectedPromote on recall: Saves a second retrieval sometimes, and undoes the saving for every one-off read.
Consequences
What it buys
  • A 50-million-object recall is one plan, not 50 million reads
  • Paid-for retrievals survive budget pauses and worker loss
  • Unread staging is visible and attributed
What it costs
  • Callers must handle job handles and callbacks
  • Staging capacity must be sized for peak recall
  • Estimates can be wrong, and are published anyway
Choose differently when
If archive reads are rare and always single objects, a simple queue with a synchronous wait of a few minutes may be enough.
Why it holds up over time
The job model is independent of media. Faster archives shorten the estimate; slower ones lengthen it; the contract with callers is the same.
LessonDo not let a temporary copy change a permanent record.
Shown on views05 20 21
ADR-27

Retrieval budgets per tenant have a hard ceiling; legal authority is never refused and never free

Accepted

Who pays for retrieval, who can see it coming, and what happens when a court order meets a budget?

Context
Absorbing retrieval centrally makes the platform's cost a function of behaviour it does not control. Charging tenants turns latency and cost into a product surface. This is an internal platform for one product, not a commercial storage service. A hard budget ceiling that blocks a legally required production is not acceptable; an unlimited legal channel becomes the way around every budget.
Decision
Retrieval is paid centrally and attributed per tenant, per reader class and per decision. Each tenant has a retrieval budget per month with a hard ceiling; exceeding it needs a named approver. A separate legal authority applies to recalls under a hold or court order: never refused for budget, approved by named counsel, charged to a legal cost line, and reviewed by security above one million objects.
How it works on-premise
Budgets and authorities are PostgreSQL rows managed through the policy service with Keycloak identities. The budget authority prices every recall before submission and pauses a running job at its ceiling with staged bytes kept.
Options weighed
  • ChosenCentral cost, tenant ceilings, separate legal authority: Protects the platform from runaway exports without ever blocking a legal obligation.
  • DeferredCharge the reading tenant directly: Right for a commercial storage product; Phase 3 tenant cost reporting prepares for it.
  • RejectedNo budgets: No friction, and one migration export can erase a month's saving.
Consequences
What it buys
  • Retrieval storms cannot become unbounded bills
  • Legal obligations never wait on finance
  • Every retrieval dollar has an owner
What it costs
  • Approver workflows add friction to legitimate large exports
  • The legal channel must be audited to stay honest
  • Budget sizing per tenant is guesswork until usage history exists
Choose differently when
If the platform is sold externally, retrieval should be a priced product feature with tenant-controlled limits.
Why it holds up over time
Separating ordinary spending authority from obligation-driven authority is an organisational pattern, not a technical one, and it stays correct whatever retrieval costs in future.
LessonGive a budget a hard ceiling, and give obligations a separate, audited door rather than a way through the ceiling.
Shown on views05 21
ADR-28

Savings are net, against an all-hot counterfactual, attributed to decisions and reconciled

Accepted

How is the saving counted so that finance believes it?

Context
A saving with no counterfactual is an assertion. A gross saving ignores retrieval, early deletion, movement requests, telemetry and the platform's own compute. The platform's own cost target (8% of net saving, about $350,000 a year) is tighter than its hardware estimate in year one.
Decision
The ledger computes daily: the counterfactual bill with every byte at the hot price; the realised bill with each byte at the price of the ladder version it was placed under; tiering charges attributed to decisions; and the platform's own cost. It publishes net saving and platform cost ratio monthly and reconciles against the storage team's chargeback within ±2%. The year-one shortfall against the 8% target is reported, not hidden.
How it works on-premise
ClickHouse queries over the catalogue replica, movement outcomes and recall records, with platform hardware and power shares loaded from the storage team's asset ledger. Reports are Grafana dashboards and a monthly signed export to finance.
Options weighed
  • ChosenNet against counterfactual, reconciled: The number survives a finance review and shows where the saving leaks.
  • RejectedCapacity freed on hot tier: Easy to show, and says nothing about the cost of freeing it.
  • RejectedGross saving by tier: Looks better every month, and hides a policy losing money.
Consequences
What it buys
  • Finance and engineering use one number
  • Leaks are attributed to policy versions and object classes
  • The platform's own cost is on the same page as the saving
What it costs
  • Reconciliation gaps must be investigated, not waved away
  • Transfer prices will be disputed
  • Year-one reports will show the platform over its cost target
Choose differently when
Nothing should flip the net and counterfactual principle. The reconciliation target moves if the storage team's own ledger is coarser.
Why it holds up over time
Counterfactual accounting outlives any tiering design: whatever replaces this platform can be compared on the same basis, which is exactly what the next decision-maker will need.
LessonReport a saving next to what would have happened without you, and after everything you cost.
Shown on views26 25

Security and operationsWho may move and who may delete, how holds change, how disagreement with storage is settled, and where it all runs.

ADR-29

Movement and deletion are different principals with fifteen-minute, one-tier credentials

Accepted

How does a platform whose whole job is copying and releasing data avoid being able to destroy it?

Context
The mover fleet is large, busy and handles every tenant's data; it is the most likely component to be compromised or buggy. If it can delete, one bad release or one stolen token removes data at the speed movement is built for. The requirement forbids long-lived storage credentials in the fleet and one principal holding both move and delete.
Decision
Movers can only PUT into one named destination tier. Only the release gate, a separate deployment under a separate identity, can DELETE, and only sources whose destination is committed and replicated. Tenant deletes go through a third identity. All credentials are short-lived: SPIRE JWT-SVIDs exchanged at RGW STS for 15-minute credentials scoped to one tier and one operation. Encryption key ids are recorded per placement and preserved by every movement.
How it works on-premise
RGW roles per tier and operation, trusting SPIRE's OIDC discovery provider. RGW SSE-KMS uses OpenBao's transit engine with a key per tenant. The release gate runs in the serving cluster with a separate Kubernetes service account, separate SPIFFE id and separate change approvals.
Options weighed
  • ChosenSeparate principals, STS from SPIFFE, one tier per role: No single identity can complete a destructive sequence, and nothing long-lived exists to steal.
  • RejectedOne storage account for the platform: Simplest, and the largest blast radius in the estate.
  • Right elsewherePer-service static RGW keys in OpenBao: Workable fallback if RGW STS with SPIRE fails in the proof, with hourly rotation.
Consequences
What it buys
  • A compromised mover can copy but not destroy
  • Stolen tokens expire in minutes
  • Key association cannot silently change during a move
What it costs
  • More roles and identities to manage
  • STS and SPIRE become dependencies of movement (not of reads)
  • Debugging access failures crosses three systems
Choose differently when
A single-tenant archive with no automated deletion could combine roles, but the separation costs so little that it rarely should.
Why it holds up over time
Separation of duties and short-lived, attested credentials are the direction of every security framework. SPIFFE and STS are standards, so storage and identity products can change without redesigning the separation.
LessonGive the busiest automation the least destructive permission, and let a small, boring service hold the dangerous one.
Shown on views28 29
ADR-30

Restrictive hold and pin changes apply at once; relaxing changes need two approvers

Accepted

Who can change holds, pins and budgets, and with what checks?

Context
Removing a hold is the one action that lets the platform move something it must not move. Placing a hold is urgent and safe: delaying it creates exposure, and it can only prevent movement. A single approval flow for both either slows holds or weakens removals.
Decision
Changes that only restrict (adding a hold, raising a pin floor, lowering a budget) apply immediately from records management or an authorised operator. Changes that relax (removing or narrowing a hold, lowering a pin floor, raising a budget ceiling) need two named approvers from different roles and are written to the Object-Locked record before they take effect.
How it works on-premise
The policy service in PostgreSQL with Patroni holds scope rows and approval records; Keycloak with WebAuthn authenticates approvers. Records-management events are idempotent by reference. Every change is written to the audit topic and Parquet archive, then applied to the catalogue with an epoch bump.
Options weighed
  • ChosenAsymmetric approval by direction of change: Fast where fast is safe, careful where careful is necessary.
  • RejectedDual approval for every change: Uniform, and delays holds during exactly the window when data is at risk.
  • RejectedSingle approval for every change: Fast, and one compromised account can release a hold.
Consequences
What it buys
  • Holds are effective within seconds
  • No single person can release a hold
  • Every relaxation is on record before it acts
What it costs
  • Two approvers must be reachable for legitimate removals
  • Classifying a change as restrictive or relaxing must itself be correct and tested
  • The approval service becomes part of the compliance scope
Choose differently when
If holds are rare and always planned days ahead, uniform dual approval is simpler and costs nothing.
Why it holds up over time
The asymmetry follows from what each change can cause, not from any product, and remains correct as long as holds exist to prevent harm.
LessonMake safe changes fast and dangerous changes deliberate, and decide which is which by what the change can cause.
Shown on views19 28
ADR-31

Reconciliation believes the tier for where bytes are and the catalogue for where they should be

Accepted

When the catalogue and storage disagree, which is right, and what happens next?

Context
Disagreement is inevitable: lost commits leave orphans, operators run tools, storage features get enabled, media fails. Absorbing drift silently hides defects. Believing only the catalogue sends reads to missing bytes; believing only storage loses the decision. Scanning nine billion objects is expensive, so frequency is a cost decision.
Decision
Bucket notifications feed near-real-time detection of changes the platform did not make. A rolling full scan covers every tier every 14 days, listing unpacked objects and reading pack manifests. Each disagreement is classified (orphan, out-of-band change, missing bytes, checksum mismatch, unknown prefix) with a fixed action per class. Unexplained drift is a defect ticket, never a silent correction.
How it works on-premise
RGW notifications to Kafka are consumed by the reconciler in the tiering cluster. Scans run as parallel list and manifest reads against Ceph, compared with the ClickHouse replica. Orphans are quarantined 35 days before the release gate deletes them.
Options weighed
  • ChosenNotifications plus 14-day scan, classified drift: Fast detection for common changes, bounded detection for everything, and every finding has a cause.
  • RejectedScan only, daily: Thorough, and roughly fourteen times the listing load for little added safety.
  • RejectedTrust the catalogue, repair on read errors: Cheap, and finds problems when a user does.
Consequences
What it buys
  • Out-of-band movement is visible within minutes
  • Orphans are cleaned without risking in-flight commits
  • Drift rates become a platform quality metric
What it costs
  • A notification missed and not yet scanned can go unnoticed for up to 14 days
  • Scan load on storage must be scheduled
  • Classification rules need maintenance as new drift causes appear
Choose differently when
If notification delivery proves unreliable in the proof phase, shorten the scan interval and price the extra listing in the ladder.
Why it holds up over time
Two sources of truth with a defined arbiter per question is a pattern that survives replacement of either source, and the 14-day interval is a parameter priced like everything else.
LessonDecide in advance which system is believed about which fact, and treat any disagreement nobody can explain as a bug.
Shown on views30 12
ADR-32

Two data centres and a witness, with cross-site failover as a rehearsed runbook

Accepted

Where does the platform run, and how does it survive the loss of a room or a site?

Context
The catalogue needs RPO 0 and a 60-second RTO in a site, and RPO 10 seconds with a 15-minute RTO across sites. Quorum systems need a third location. Automatic cross-site failover of a database on every product read path risks a split brain worse than a short outage. The storage tiers already replicate between the two data centres.
Decision
DC-A is primary, with three rooms as independent fault domains. Vitess primaries and two semi-sync replicas sit in different rooms, with VTOrc failover inside DC-A. DC-B holds async replicas, warm resolvers and the second Ceph zone and tape library. A small witness site holds the third etcd voter and the dead-man monitor. Promoting DC-B is a human-initiated, rehearsed runbook.
How it works on-premise
Kubernetes (RKE2) with Cilium in each data centre, separate clusters for serving and tiering. Rooms are exposed to schedulers as topology zones. The DC-B promotion runbook is rehearsed quarterly with a measured RTO recorded against the 15-minute target.
Options weighed
  • ChosenPrimary with rooms, warm secondary, witness, manual site failover: Automatic where the failure is local and clear, deliberate where it is not.
  • RejectedActive-active across sites: No site failover at all, and cross-site semi-sync on every commit plus conflict risk on the catalogue.
  • RejectedAutomatic cross-site failover: Faster RTO, and a network partition can promote DC-B while DC-A still serves.
Consequences
What it buys
  • Room failures heal without people
  • No automated path to a split-brain catalogue
  • The witness keeps quorum and monitoring independent of both sites
What it costs
  • A site loss is a 15-minute outage and needs an operator
  • DC-B hardware is mostly idle
  • Quarterly rehearsals take real time
Choose differently when
With three or more well-connected data centres, run Vitess primaries per region with cross-site semi-sync and automate site failover.
Why it holds up over time
Fault domains, quorum with a witness and human judgment for ambiguous site failures are stable patterns that do not depend on these products. Measured rehearsals keep the RTO a fact rather than a hope as the estate changes.
LessonAutomate recovery from failures you can diagnose for certain, and rehearse the ones you cannot.
Shown on views22 31

Every package used, in one table

Every open-source package named on the views, what it is, the job it does here, and what was considered instead.

PackageWhat it isWhat it does hereConsidered instead
Vitess Sharding and clustering system for MySQL, CNCF graduated Placement catalogue routing, sharding, failover (VTOrc) and change data (VStream) TiDB, FoundationDB
MySQL 8.4 Relational database, long-term support release Storage engine under every catalogue shard PostgreSQL with Citus
Ceph RGW S3-compatible object gateway over Ceph RADOS Hot, warm and cold tiers, staging, records archive, STS and SSE-KMS MinIO, SeaweedFS
EOS Disk storage system from CERN Disk buffer in front of the tape archive dCache
CERN Tape Archive (CTA) Tape archive and retrieval scheduler from CERN Tape libraries, drives, retrieval queues, repack, two tape copies LTFS with own scheduler
Apache Kafka Distributed log, Apache Software Foundation Access events, decisions, movement plans and outcomes, CDC, bucket notifications Redpanda, NATS JetStream
ClickHouse Columnar analytical database Access aggregates, analytical catalogue replica, records query, cost ledger Apache Druid, Trino on Iceberg
Temporal Durable workflow engine Policy rollout rings, recall jobs, offboarding Argo Workflows
PostgreSQL with Patroni Relational database with HA manager Policy service, holds, pins, budgets, Temporal persistence MySQL with Orchestrator
CUE Configuration language with types and constraints Tier ladder and placement policies OPA Rego, JSON Schema
Forgejo Self-hosted Git forge Policy and ladder repositories, reviews, CI GitLab CE
cosign Artifact signing tool, Sigstore Signs ladder and policy bundles GPG signatures
SPIRE SPIFFE workload identity implementation, CNCF graduated Attested identities for movers, release gate and resolvers; OIDC for RGW STS Kubernetes service account tokens
OpenBao Secrets and key management, Linux Foundation fork of Vault Per-tenant encryption keys for RGW SSE-KMS HashiCorp Vault
Keycloak Identity and access management, CNCF incubating Operator and approver authentication with WebAuthn Dex, Authentik
Kubernetes (RKE2) Container orchestration distribution Serving and tiering clusters OKD, kubeadm
Cilium eBPF networking and policy for Kubernetes Network policy between serving and tiering workloads Calico
etcd Consistent key-value store, CNCF graduated Vitess topology with a voter at the witness site ZooKeeper
Prometheus, Thanos, Alertmanager Metrics, long-term storage and alerting SLOs, freshness, budgets, ring gates VictoriaMetrics
Grafana Dashboards Operations and cost ledger views Apache Superset
OpenTelemetry and Tempo Tracing standard and trace store Slow resolves, movement batches, recall jobs Jaeger
Healthchecks Self-hosted dead-man monitoring External watcher for the telemetry canary at the witness site Alertmanager watchdog
LightGBM Gradient-boosting library Phase 2 predictive classifier, in shadow first XGBoost
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises31 views · HTML / SVG / draw.io