Architecture Decision Record

Storage Tiering Service · Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises

The argument these decisions serve is summarised in the Architecture One-Pager.

Thirty-two decisions make up this architecture. Everything else on the thirty-one views is convention, and convention needs no defence. Each record opens with the question that forced a decision, says what was chosen and how it is built on hardware the organisation owns, lists the options that lost and why, and states what the choice costs. It then names the conditions under which a different organisation should choose differently, and explains why the choice should still be right after the products, the media and the team have changed. Read the one-pager first. Read a record when you want to argue with it.

Status of this document. This is a design, not a report on a running system. Prices, latencies, volumes and savings are targets or stated assumptions taken from the requirement or from planning estimates, and each is replaced by a measured value in the proof phase. Five product behaviours are relied on and must be confirmed before production depends on them: Ceph RGW's STS accepting SPIRE-issued JWT-SVIDs through AssumeRoleWithWebIdentity; RGW validating S3 additional checksums (CRC32C) on PUT in the deployed Ceph release; RGW bucket notifications delivering object and lifecycle events to Kafka without silent loss under load; EOS and CTA serving 2,000 concurrent stage requests through the WLCG Tape REST API; and Vitess VStream keeping a 9.2-billion-row analytical replica current while the catalogue takes 1,400 commits a second.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it works on-premise: The concrete mechanism: which package, configured how, on whose hardware.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

Read path and catalogue: What a read is allowed to depend on, where placement truth lives, and why stale answers are safe.

• ADR-01 · The read path never consults the tiering plane
• ADR-02 · The placement catalogue runs on Vitess over MySQL, sharded by tenant
• ADR-03 · Bounded-stale resolution, made safe by a 24-hour release hold
• ADR-04 · The catalogue decides, the tier is fact, and both can be rebuilt from storage
• ADR-05 · Callers see a recall class and a job handle, never a tier

Tiers and storage: What a tier is when the organisation owns the hardware, which software provides each rung, and how small objects are stored.

• ADR-06 · The tier ladder is signed, versioned configuration priced in internal transfer prices
• ADR-07 · Ceph RGW provides the disk tiers, one cluster set per tier, with lifecycle transitions disabled
• ADR-08 · Small objects are stored in single-tenant, single-key, self-describing packs
• ADR-09 · Compact a pack only when dead space passes half and minimum durations have expired
• ADR-10 · The archive rung is LTO tape behind EOS and CTA, driven through the WLCG Tape REST API
• ADR-11 · The ladder ends in a second custodian, in Phase 3, only for data kept as an obligation

Signal and classification: What counts as evidence of use, how it is collected cheaply, who classifies, and what happens when the signal goes quiet.

• ADR-12 · Every reader declares a class, and non-promoting classes change nothing
• ADR-13 · Telemetry is reduced at the source and stored in Kafka and ClickHouse
• ADR-14 · Classification is a bounded batch over an analytical replica of the catalogue
• ADR-15 · Declarative policy first; a predictive classifier earns its place in shadow
• ADR-16 · The platform classifies; the storage layer does not tier itself
• ADR-17 · Objects under 1 MB are classified and moved only as cohorts
• ADR-18 · Blind means no demotion: one canary drives both the brake and the alarm

Placement policy: The arithmetic every demotion must clear, what stops oscillation, where guards are enforced, and how a policy reaches production.

• ADR-19 · Every demotion clears a per-object break-even over the destination's minimum duration, recorded with its inputs
• ADR-20 · Hysteresis: 14-day dwell, a 3× promotion bar, and no promotion on a first read
• ADR-21 · Guards are enforced at commit against a per-tenant epoch, not at planning
• ADR-22 · Policy ships through shadow, dry run, rings and a hard movement cap, orchestrated by Temporal

Movement: How an object changes place without ever being lost, who does the work, and what a delete does to a move in progress.

• ADR-23 · Copy, verify, commit, release, with the catalogue row as the state machine
• ADR-24 · Movers are a Go fleet on budget leases that name dollars and requests
• ADR-25 · A delete wins over a movement, and offboarding is proven against storage

Recall and cost: How cold data comes back, who pays for it, and how the saving is counted so that it survives a finance review.

• ADR-26 · Recall is a Temporal job planned by pack and cartridge, and staging is not promotion
• ADR-27 · Retrieval budgets per tenant have a hard ceiling; legal authority is never refused and never free
• ADR-28 · Savings are net, against an all-hot counterfactual, attributed to decisions and reconciled

Security and operations: Who may move and who may delete, how holds change, how disagreement with storage is settled, and where it all runs.

• ADR-29 · Movement and deletion are different principals with fifteen-minute, one-tier credentials
• ADR-30 · Restrictive hold and pin changes apply at once; relaxing changes need two approvers
• ADR-31 · Reconciliation believes the tier for where bytes are and the catalogue for where they should be
• ADR-32 · Two data centres and a witness, with cross-site failover as a rehearsed runbook

Technology by capability

Every capability on the views, the package that provides it, a credible alternative, and the record that justifies the choice. Everything runs on hardware the organisation owns, with no managed service and no external dependency on any read path.

The decisions, and the alternatives that lost

Read path and catalogue

What a read is allowed to depend on, where placement truth lives, and why stale answers are safe.

ADR-01 · The read path never consults the tiering plane

Status: Accepted · Shown on views: 02, 07, 08, 11

What is a read allowed to depend on?

Context. Every product read has to find its object, so whatever answers that question sets a ceiling on the product's availability and a floor on its latency. The tiering plane (telemetry, classifier, planner, movers) is large, batch-shaped and changes weekly. If a read ever calls it, even as a fallback, the storage bill and the product's availability become the same number.

Decision. Placement resolution is a lookup in the committed catalogue and nothing else. It does not evaluate policy, call the classifier, check a hold or probe a tier. The serving plane (resolver, read router, placement API, recall API, catalogue) is a separate Kubernetes cluster with a 99.99% target; the tiering plane is another cluster with a 99.5% target and no component the serving plane calls.

How it works on-premise. The resolver is a Go service whose only network dependency is vtgate. Its deployment manifests have no service entries for Kafka, ClickHouse, Temporal or the policy service, and a CI check fails any change that adds one. Access events are written to a local bounded buffer and shipped asynchronously; if Kafka is down the buffer drops events, never reads.

What it buys

• Classifier, planner and mover outages are cost events, never availability events
• The tiering plane can be redeployed, rewritten or paused without a change window for reads
• The read path is small enough to reason about completely

What it costs

• A cold read cannot promote itself synchronously; promotion waits for the next cycle
• Every read emits telemetry through a buffer that must be sized and watched
• The catalogue becomes the one component whose availability everything inherits

Choose differently when. A system where every tier is inside one storage product that already tracks class per object, and where the product's own metadata service is the read path anyway, gains nothing from a separate catalogue and should use the product's placement.

Why it holds up over time. The boundary is about dependency direction, not technology. Whatever replaces the classifier, the movers or even the storage tiers, the rule that reads depend only on committed placement stays correct and keeps the replacement low-risk.

Lesson. Put the system that decides on a different availability budget from the system that serves, and forbid the call between them rather than hoping nobody makes it.

ADR-02 · The placement catalogue runs on Vitess over MySQL, sharded by tenant

Status: Accepted · Shown on views: 08, 13, 22

Which open-source store can hold 9.2 billion placements, answer point reads at 1.4 million a second, and commit a conditional update with no data loss inside a data centre?

Context. The requirement asks for strong consistency, RPO 0 in a data centre, no cross-tenant transactions, 3.7 TB of rows before indexes, 34% growth a year, and a p99 of 8 ms. Every operation is either a point read by key or a single-row or single-tenant conditional write. There are no joins across tenants, no range scans on the hot path and no analytical queries, because those go to a replica (ADR-14).

Decision. Vitess on MySQL 8.4, 64 shards to start, keyed by a multicolumn vindex of tenant hash and object hash. Each shard has a primary and two semi-sync replicas in separate rooms of DC-A, and one asynchronous replica in DC-B. VTOrc handles in-DC failover. Commits are compare-and-set updates on row_version and guard_epoch within one shard.

How it works on-premise. MySQL runs on bare-metal NVMe servers, not in Kubernetes, managed by the Vitess operator's topology in etcd with the third etcd voter at the witness site. vtgate runs in the serving cluster. Resharding is done online with MoveTables and Reshard workflows as tenants grow. Backups use Vitess's built-in backup to a Ceph bucket that is not a placement tier.

What it buys

• Single-shard transactions are exactly the shape of every commit
• Semi-sync across rooms gives RPO 0 in DC-A with plain MySQL operations knowledge
• Online resharding lets the catalogue grow with the corpus without a migration project

What it costs

• The shard key is a design decision the team owns forever
• 256 MySQL instances is a real fleet to patch and watch
• Cross-site failover is a runbook rather than automatic, by choice (ADR-32)

Choose differently when. Choose TiDB when the team cannot own a sharding scheme and point-read cost is not the constraint. Choose FoundationDB when cross-key transactions become necessary, for example if placements start spanning tenants.

Why it holds up over time. The catalogue's access pattern is key-value with conditional writes, so its schema moves to any store with that capability. MySQL's replication and storage engine have been stable for fifteen years, and Vitess's governance under the CNCF does not depend on one company's roadmap.

Lesson. Pick the store for the operations you actually run on the hot path, and send everything else to a replica built for it.

ADR-03 · Bounded-stale resolution, made safe by a 24-hour release hold

Status: Accepted · Shown on views: 11, 18, 31

How can resolution use caches and replicas when the requirement says placement must be strongly consistent?

Context. 1.4 million resolutions a second against primaries alone would need a much larger catalogue fleet, and the 4× burst would need more still. Caches and replicas are the obvious answer, and the obvious danger is a stale answer pointing at a location whose bytes were just deleted. The property that actually matters is not linearisable reads. It is that a resolved location always holds the bytes.

Decision. Sources are released no earlier than 24 hours after commit. Resolution may therefore use a 60-second in-process cache and replicas lagging less than 5 seconds, and falls back to the primary only when a row is not found. Commits remain serialisable on the primary. The design guarantees that every answer, however stale within 24 hours, names bytes that exist.

How it works on-premise. The movement row carries release_after = commit_time + 24 h. The release gate selects only rows past that time. vtgate routes resolver reads to replicas with a lag threshold of 5 seconds and to the primary on not-found. The resolver cache is a bounded LRU with a 60-second TTL and no negative caching.

What it buys

• Most resolutions never reach a primary
• Reversing a bad movement within a day is a pointer change with no retrieval
• Long streams that started before a commit finish from the source

What it costs

• One extra day of storage for every moved byte, about \(250 to \)600 a day at planned volumes
• A newly committed placement may be read at its old location for up to 60 seconds
• The guarantee depends on the release gate honouring release_after, so the gate is tested harder than anything else

Choose differently when. Where storage is so expensive that a day of duplicates matters, or where movements are rare enough that primaries can serve every read, drop the hold and read from primaries.

Why it holds up over time. This is a property of ordering: no cache, replica or storage technology can break it as long as release waits longer than the longest staleness. Future caches or read paths only need to state their staleness bound and check it is under 24 hours.

Lesson. When readers can be stale, make the writer wait out the staleness before destroying anything they might still point at.

ADR-04 · The catalogue decides, the tier is fact, and both can be rebuilt from storage

Status: Accepted · Shown on views: 12, 30, 31

Where does placement truth live: in the catalogue, in the storage layer's own metadata, or in both?

Context. A catalogue-only design is fast and loses everything if the catalogue is lost. A storage-only design is rebuildable and puts a listing over nine billion objects on the read path. The requirement wants catalogue reads on the hot path, a rebuild within 30 hours, and drift detected rather than absorbed.

Decision. The catalogue is authoritative for where to read. The storage tiers are believed about where bytes physically are. The two are reconciled continuously (ADR-31). Everything needed to rebuild location is written into storage itself: object keys encode tenant, object and version, and every pack carries its own index plus a sidecar manifest.

How it works on-premise. Unpacked keys follow t//