Storage Tiering Service  ·  View 20 of 31  ·  5 · Runtime

Recall From Tape

A request for 1.8 million objects becomes one job with a price, an authority, an honest estimate and a report of what was never read.

Editable source SVG draw.io All views
Legal operations Recall API Catalogue Budget authority Recall workflow EOS + CTA Staging bucket 1. recall owner j.doe · 2019–2025 2. resolve 1.8 M placements 3. 1.1 M immediate · 0.7 M delayed 4. price 2,300 packs on 31 cartridges 5. over tenant ceiling · legal authority 6. job J-88 · ETA 31 h · approver needed 7. approved by counsel 8. start J-88 9. stage 2,300 packs · Tape REST 10. order by cartridge and position 11. packs staged, in batches 12. unpack members · TTL 7 d 13. ETA revised to 36 h 14. ready · unread bytes reported at expiry Recall From Tape — A Job, Not a Hung Request Staging is not promotion: the placement rows still say archive, and the staged copies expire unless a policy decides otherwise. v 1.0 · owner Storage Platform Architecture · date 2026-09

Decisions

  • Recall is priced before it is authorised. The workflow resolves placements, groups them into packs and cartridges, and shows the requester the cost and the estimate before any drive moves.
  • The recall workflow runs in Temporal because a job can outlive worker restarts, budget pauses and a 12-hour tape queue. The state that has already cost money is exactly the state that must not be lost.
  • Rehydration is not promotion. Staged copies live in a TTL bucket and expire; the placement still says archive unless a promotion policy decides otherwise (ADR-26).

Numbers

  • Archive rehydration p50 4 min, p99 45 min for a single object; 12-hour ceiling declared to callers. Staging TTL 48 hours by default, extendable per job to 7 days.
  • Unread rehydrations target at most 3% of rehydrated bytes. Every staged-but-unread byte is charged to the job's authority.

Risk

  • The design relies on EOS and CTA exposing staging through the WLCG Tape REST API at this request volume. The proof phase tests 2,000 concurrent stage requests; the fallback is CTA's native frontend.
◀  A Hold and a Delete Arrive Mid-Move All views Retrieval Storm  ▶
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises20-archive-recall-job