GPU Fleet & Capacity
Scheduling and quota, fragmentation, preemption, multi-tenancy, and planning capacity under lumpy demand.
5concepts
62flashcards
36minutes of reading
- 01 Capacity Planning Under Lumpy Demand Why GPU demand does not smooth the way CPU demand does, how to reason about the reserved-versus-on-demand mix, and the lead times that make this a quarters-ahead decision.
- 02 Fault Tolerance for Long Training Runs Why failure is the expected case at scale, the arithmetic that sets checkpoint frequency, and the detection problem that makes silent corruption worse than a crash.
- 03 Gang Scheduling and Fragmentation Why a distributed training job cannot start until every worker starts, how that requirement produces both deadlock and stranded capacity, and the placement constraints that make a cluster's usable size smaller than its size.
- 04 Multi-Tenancy and GPU Sharing Why one job per GPU wastes most of the hardware for small models, the three sharing mechanisms and their isolation guarantees, and the interference that makes sharing unsuitable for latency-sensitive work.
- 05 Preemption, Priority and Spot Capacity How interruptible capacity changes the cost of training by a large factor, what a job must be able to do to use it, and the failure modes that make spot capacity a false economy for the unprepared.