Portfolio ↗

Accelerator Architecture

The memory wall, roofline analysis, GPU execution model, interconnects and systolic arrays.

20concepts
140flashcards
158minutes of reading
Start with Collective Communication Primitives Drill the deck
All levels beginner intermediate advanced
  1. 01 Hardware Cost of Mixture-of-Experts Sparse MoE models reduce FLOPs per token but introduce all-to-all communication, load-imbalance penalties, and memory pressure that can erase those savings unless the system is carefully co-designed. advanced 9m 7 cards