Storage Tiering Service  ·  View 18 of 31  ·  5 · Runtime

One Movement: Copy, Verify, Commit, Release

The four steps in order, the two conditions a commit checks, and the three conditions a release checks, for one object going from warm to cold.

Editable source SVG draw.io All views
Mover worker Budget authority Warm tier · DC-A Cold tier · DC-A Catalogue Release gate Cold tier · DC-B 1. lease 40 GB · 5,000 requests 2. granted for 5 min 3. mark in-flight m-77 if row_version 12 4. ok · guard_epoch 9 5. GET object 6. PUT with CRC32C 7. stored · checksum matches 8. commit if epoch 9 and not tombstoned 9. committed · release_after +24 h 10. releases due on shard 17 11. HEAD destination replica 12. present · checksum ok 13. DELETE source copy 14. mark released One Movement — Copy, Verify, Commit, Release Until message 9 the object reads from warm. After it, reads go to cold while warm stays intact for 24 h as a free undo. v 1.0 · owner Storage Platform Architecture · date 2026-09

Decisions

  • The movement row in the catalogue is the state machine. A worker that dies at any step leaves a row saying exactly how far it got, and the next worker resumes or abandons from there (ADR-23).
  • The commit is a compare-and-set: it succeeds only if the tenant's guard epoch is unchanged and the object is not tombstoned. The destination write alone changes nothing.
  • Release is done by a different service under a different identity, 24 hours after commit, and only after the destination's replica in DC-B is confirmed. Otherwise a site loss in the replication gap could remove the only surviving copy.

Numbers

  • Budget leases are for five minutes and name both bytes and requests. At this object-size distribution the request count, not the byte count, is what runs out first.
  • Holding sources for 24 hours costs one extra day of storage for 900 TB: about $250 a day from warm, $600 from hot. It buys a free undo for every movement.

Risk

  • The checksum is computed by the mover and validated by RGW on receipt. That depends on RGW honouring S3 additional checksums on the deployed Ceph release; if it does not, verification falls back to a read-back, which doubles read load on the destination.
◀  The Break-Even Inequality All views A Hold and a Delete Arrive Mid-Move  ▶
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises18-movement-sequence