# Architecture One-Pager

*Storage Tiering Service · Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises*

**Every demotion is a bet that an object will not be needed before the saving pays for the move. This architecture makes each bet explicit and priced, keeps every bet off the read path, and makes sure that a losing bet costs money and never costs an object.**

A file-collaboration product keeps everything its customers ever upload: 9.2 billion objects and 41 PB today, growing 34% a year. Most of it is read heavily for a few weeks and then almost never, but almost never is not never, and nobody can say in advance which old file a customer will open tomorrow. Left alone, the storage bill grows with everything ever stored. Tiered carelessly, it grows anyway, through retrieval charges, early-deletion charges, movement costs and a telemetry pipeline that can cost more than it saves, while reads that used to take 40 milliseconds start taking four minutes. The hard parts are not the storage tiers. They are knowing when absence of access really means coldness, pricing each move honestly, moving nine billion objects without ever holding the only copy somewhere unrecorded, and making sure that the machinery doing all of this can fail completely without a single read failing with it.

Every read resolves its object through a placement resolver: a stateless Go service with a 60-second cache in front of a Vitess catalogue on MySQL, sharded by tenant. The resolver returns a location, a byte range and a recall class, and the file service reads the bytes directly from Ceph RGW. Hot, warm and cold tiers are separate Ceph clusters; objects under 1 MB are stored in cold as single-tenant packs; archive is LTO tape behind EOS and the CERN Tape Archive. Each resolved read emits a deduplicated access event into Kafka, and ClickHouse keeps per-object and per-cohort aggregates. Every six hours a classification cycle reads those aggregates and a change-data replica of the catalogue, evaluates each candidate's observation window and break-even against the signed tier ladder, and records every decision with its inputs. A Temporal workflow rolls a policy out through shadow, dry run and rings. Go movers copy each object, verify its checksum, and commit the new placement with a compare-and-set that fails if a hold arrived or the object was deleted. Twenty-four hours later a separate release gate, holding the only delete permission, removes the source once the destination is confirmed in the second data centre. Recalls from tape are Temporal jobs planned by pack and cartridge, charged against budgets and staged with a time limit. Every charge is attributed to the decision that caused it and reported against what the corpus would have cost if it had all stayed hot.

## What it is, and what it is not

- **A catalogue lookup on every read** — not a read path that asks a classifier, a policy engine or a storage tier where an object went.
- **Placement decisions recorded with their odds and their price** — not lifecycle rules that move objects nobody can later account for.
- **A saving counted net, against an all-hot counterfactual** — not a gross storage delta that leaves out retrieval, early deletion, movement and the platform's own cost.
- **Recall classes with published estimates** — not tier names, vendor classes or hung requests leaking into the product.
- **A service that places objects into storage it does not run** — not a storage system, a deletion policy or an encryption scheme.
- **Tiering for objects that pay for it** — not a platform that moves 240 KB photos one at a time at a loss.

## The decisions that are the architecture

1. **The read path never asks the tiering plane** (ADR-01) — Resolution is a catalogue lookup and nothing else. The classifier, planner and movers carry 99.5% availability; placement resolution carries 99.99%, and the first can be switched off without touching the second.
2. **Stale answers are safe because sources outlive them** (ADR-03) — A source copy is released 24 hours after commit, which is longer than any cache, replica lag or stream. Resolution can use caches and replicas, and every movement has a free undo for a day.
3. **A movement is a catalogue state machine** (ADR-23) — Copy, verify, commit, release, with the in-flight state in the catalogue row. A worker that dies anywhere leaves a row that says what to do next, and the source is never released before the commit.
4. **A hold is an epoch, checked inside the commit** (ADR-21) — Holds are scope rows beside the tenant's placements. Placing one bumps a per-tenant epoch, and every movement planned under the old epoch fails to commit. No row in nine billion needs touching.
5. **Every demotion clears a recorded break-even** (ADR-19) — Saving over the destination's minimum duration must exceed expected retrieval, movement and early-deletion cost. The inputs, the odds and the verdict are stored, so every decision can be scored.
6. **Small objects move as cohorts in packs** (ADR-17) — Below 1 MB the per-object overhead exceeds the saving. Those objects are classified by folder and moved only as members of a single-tenant pack, or not at all.
7. **Blind means no demotion** (ADR-18) — One canary per telemetry partition drives both the freshness gate and an external dead-man monitor. When the signal is stale, demotion stops and promotion, recall and reads carry on.
8. **Moving and deleting are different identities** (ADR-29) — Movers can only write into one tier with 15-minute credentials. Only the release gate can delete, and only a source whose destination is committed and replicated.

## Why this holds up over time

Tiering platforms tend to fail slowly rather than suddenly. Prices and media change and the hard-coded policy quietly becomes a money-losing one. The classifier is replaced and nobody can say whether the new one is better. The object count grows until per-object telemetry and requests cost more than the tiers save. The storage product or the tape generation changes and the platform turns out to depend on it. The engineers who knew why objects are where they are move on. Or a core component changes its licence. The design addresses each of these deliberately.

- **Prices are data, not code.** The ladder is signed, versioned configuration, and every placement records the version it was made under. A new tape generation, a cheaper disk or a retired tier is a ladder change evaluated in shadow, not a code change, and existing placements stay valid under the version that made them.
- **Decisions carry their own evidence.** Each decision stores its inputs, its predicted read probability and its break-even date, and each charge is attributed to one. A future team replacing the classifier compares the two on realised money, not on opinion, and does not need anyone from this team in the room.
- **Work grows with bytes, not with object count.** Telemetry is deduplicated per object-hour, small objects are handled as cohorts, and packs turn millions of requests into one. Doubling the corpus to 18 billion mostly small objects adds catalogue rows but little telemetry and very few movements.
- **Storage sits behind standard interfaces.** The S3 API and STS for disk tiers, the WLCG Tape REST API for tape. Ceph can be replaced by another S3 store and EOS with CTA by dCache without changing the placement model. LTO generations roll through CTA's repack, invisible to the catalogue except as a location change.
- **Every object can be found without this software.** Object keys encode tenant, object and version, and every pack carries its own index and a sidecar manifest. If the catalogue and everyone who built it were gone, a script and the tiers are enough to recover where every object is.
- **Correctness comes from ordering, not from a product.** Copy, verify, commit, release and an epoch-checked commit need only a store with conditional writes and an object store with PUT and DELETE. The safety argument survives a change of database and a change of storage.
- **Governance risk is named, with an exit for each.** Vitess, Kubernetes, SPIRE and OpenBao sit under neutral foundations. Where one company leads, the exit is named: ClickHouse to Parquet with Trino, Temporal to PostgreSQL-backed workers, CTA to dCache or plain LTFS. CockroachDB and MinIO were considered and set aside after their licensing and community-edition changes showed what that risk looks like.
- **The platform measures itself.** Net saving, platform cost as a share of it, per-object overhead and the smallest object worth tiering are published every month. Whoever re-decides these choices starts from those numbers rather than from this document.

## Non-functional targets

Targets from the requirement, and the mechanism that meets each. Every latency, throughput and cost figure is replaced by a measured value in the proof phase and republished monthly.

| Quality | Target | How it is met | View |
|---|---|---|---|
| Placement resolution availability | ≥ 99.99% monthly | Stateless resolvers in three rooms; Vitess semi-sync with VTOrc failover; nothing from the tiering plane on the path | 08 |
| Placement resolution latency | p50 ≤ 2 ms · p99 ≤ 8 ms | 60 s in-process cache, lag-bounded replica reads, primary only for rows not yet replicated | 11 |
| First byte by tier | p99 45 ms hot · 120 ms warm · 900 ms cold | NVMe replicas, HDD erasure coding, ranged GET into cold packs without reconstructing the pack | 09 |
| Archive rehydration | p50 ≤ 4 min · p99 ≤ 45 min · 12 h ceiling | Two drives reserved for interactive recalls; CTA orders stages by cartridge position; estimates published per tier | 21 |
| Synchronous ceiling | ≤ 3 s, then a job handle | Recall class known at resolution; delayed and scheduled objects return a handle and never hold the request | 11 |
| Read throughput | 1.4 M resolutions/s · 4× burst for 60 s | 48 resolver pods sized on cache hits; catalogue replicas sized to survive a 50% cache hit rate | 08 |
| Movement capacity | 900 TB and 120 M objects/day · 4× backfill | 24 mover nodes on budget leases; small objects moved as packs; backfills yield to steady-state work | 18 |
| Bulk recall | 50 M objects in one job | Temporal job planned by pack and cartridge, quota-aware, resumable across budget pauses | 20 |
| Classification cycle | ≤ 6 h over the whole corpus | ClickHouse SQL over the CDC replica and aggregates; break-even arithmetic in Go on the candidate set only | 16 |
| Net storage cost | ≤ $0.0104/GB-month · ≥ 45% below all-hot | Break-even per decision, cohorts for small objects, ingest placement, net saving reconciled against chargeback | 26 |
| Tiering charges | ≤ 6% of gross saving · demotion pauses at 12% | Minimum duration as the bet horizon, 14-day dwell, 3× promotion bar, ring gates on charge ratio | 23 |
| Read quality | Wrong-tier reads ≤ 0.05% · unread rehydration ≤ 3% | Periodic classes with 400-day windows, drive reservations, staging TTL and unread-byte attribution | 25 |
| Object data RPO | 0 | Copy, verify, commit, release; source released 24 h after commit and only after the destination replica is confirmed | 18 |
| Catalogue recovery | RPO 0 · RTO 60 s in DC · RPO 10 s · RTO 15 min across sites | Semi-sync replicas in separate rooms; async replicas and warm resolvers in DC-B; rehearsed promotion runbook | 22 |
| Catalogue rebuild | ≤ 30 h, location only | Self-describing keys and pack manifests scanned in parallel; history restored from the records store | 30 |

## Scope

**In scope**

- Placement resolution and the recall class contract for every object read
- The placement catalogue, its analytical replica and its rebuild by tier scan
- Access telemetry with reader classes, deduplication and a freshness gate
- Declarative classification with per-object break-even, windows, dwell and promotion rules
- Policy delivery through shadow, dry run, rings and a movement cap
- Copy, verify, commit and release across Ceph tiers and tape, with packing and compaction
- Recall jobs, staging, retrieval budgets and legal authority
- Holds and pins as constraints, reconciliation and drift, and the net cost ledger

**Explicitly out of scope**

- Operating Ceph and the tape estate, and the durability each tier declares
- Deciding what may be deleted, retention schedules and determining legal holds
- Serving bytes to end users, client and edge caching
- Deduplication, compression and the choice of encryption at rest
- Tenant-facing cost reporting and self-service policy, which are Phase 3

## The ten-week proof

The proof does not try out features. It tests the properties everything else depends on: the read path at full load, a movement protocol that cannot lose an object, a saving that exists on real access logs, and the five product behaviours the design takes on trust.

1. Resolver and a 16-shard Vitess catalogue loaded with 2 billion synthetic placements, driven at 1.4 M resolutions a second with a 4× burst
2. Movers, release gate and reconciler against two small Ceph clusters in two rooms, with RGW STS and SPIRE credentials
3. Shadow classification over 90 days of real access logs from the current product, producing a projected net saving
4. One EOS and CTA instance with a small library, driven through the Tape REST API at 2,000 concurrent stage requests
5. A VStream replica into ClickHouse kept current under 1,400 commits a second

- Run 10 million movements while killing workers, dropping commit acknowledgements and restarting shard primaries; then prove by scan that every object is readable and exactly one copy is named
- Place a hold on a tenant with 50,000 movements in flight; prove that none of them commits and none of their sources is released
- Stop the telemetry pipeline; prove that demotion stops within 15 minutes, promotion and recall continue, and the external monitor pages
- Cut DC-A in the middle of a release batch; prove that no object loses its last copy

## Open risks, carried rather than hidden

| Risk | If it lands | Response |
|---|---|---|
| The requirement's figures do not agree with each other | 1.4 M reads a second does not fit 120 B events a month without deduplication; 210 TB a day of ingest does not fit 34% growth without heavy deletion; the per-object overhead target is thirteen times looser than the 8% cost ceiling | Deduplicate at source (ADR-13), treat early deletion as a first-class input (ADR-19), and hold the platform to the tighter 8% figure, reported monthly |
| The platform costs more than 8% of the saving in year one | Hardware alone is estimated at $450,000 to $600,000 a year against a ceiling of about $350,000 | State it; the ratio is met in year two or three as the corpus grows against a mostly fixed platform. If the proof shows otherwise, buy less tiering: defer tape and packs (ADR-28) |
| Five product behaviours are unconfirmed | STS with SPIRE, RGW checksums, notification reliability, Tape REST at volume or VStream at scale may not behave as assumed | Each is a named proof test with a fallback: per-tier RGW keys rotated hourly from OpenBao, read-back verification, shorter scan interval, CTA native frontend, periodic snapshot export |
| The catalogue is on every product read | A catalogue incident is a product outage, and its latency is a floor on every read | Fewest features of any component, semi-sync across rooms, warm DC-B, 24-hour release hold so stale caches still serve; the catalogue gets the platform's strictest change process (ADR-02, ADR-03) |
| Objects under 1 MB stay hot through the MVP | About 7% of the target saving is deferred to Phase 2 | Accepted: they are 75% of objects and 4% of bytes, and moving them one at a time would lose money (ADR-17) |
| Tape skills and CTA experience are scarce outside research computing | Slow incident response on the archive rung; hiring difficulty | Archive is Phase 2, behind a standard API; the storage team runs it; LTFS and dCache are named alternatives if CTA does not fit (ADR-10) |

The reasoning behind every component and technology choice is in the [Architecture Decision Record](decision-record): 32 records across 7 areas, each with the alternatives that lost and what the choice costs.