Spot Capacity Orchestrator  ·  View 20 of 32  ·  5 · Runtime

Checkpoint and Restore

Periodic checkpoints as insurance, a final write on notice, a manifest that makes a generation real, and a resume that cannot emit its result twice.

Editable source SVG draw.io All views
Workload Checkpoint client Node agent Ceph RGW bucket Scheduler Restored workload 1. save() every 60 min 2. gen-17 parts, SSE-KMS 3. manifest gen-17 written last 4. notice · deadline T+120 5. on_notice() 6. save(1.4 GB) 7. gen-18 parts + manifest 8. committed in 31 s 9. checkpoint committed 10. if RGW unreachable: release lease, no resume 11. reschedule unit 12. latest manifest 13. gen-18 14. resume · outputs staged until done Checkpoint and Restore — Periodic Insurance, Final Write, Clean Resume A generation without its manifest does not exist. Outputs are published only when the unit completes, so a resumed unit cannot emit a result twice. v 1.0 · owner Compute Platform Architecture · date 2026-09

Decisions

  • Application checkpoints through a platform client, not process snapshots. The client handles encryption, multipart upload, the manifest commit and the notice callback; the workload decides what state is.
  • A generation exists only when its manifest is written, last. A write interrupted halfway leaves parts without a manifest, which restore ignores and lifecycle rules delete.
  • Restart semantics are declared: outputs go to a staging prefix and are published when the unit completes, so a resumed unit cannot produce a duplicate externally visible effect.

Why periodic checkpoints can be infrequent

  • With notice, most reclaims end in a final write. Only no-notice terminations (2%), overruns (0.5%) and failed final writes (about 1%) lose work since the last periodic checkpoint: roughly 0.14% of instances an hour instead of 4%.
  • The Young–Daly interval √(2 × cost × MTBF) with a 45 s checkpoint and a 714-hour effective MTBF is about 4 hours. The default is capped at 60 minutes, because the no-notice rate is an assumption and one lost hour is what an ML engineer said they could accept.

Store unavailable

  • If RGW cannot be reached during drain, the client releases the work item's lease and the agent records the failure. The unit restarts from its last committed generation; nothing is lost silently.
◀  Four Drain Paths All views Correlated Reclamation  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises20-checkpoint-and-restore