Inference & Serving
29 min
Arithmetic Intensity: Why Your GPU Is Idle 99% of the Time
An H100 advertises 989 teraflops. Generating one token from an 8B model uses roughly 0.3% of that. The gap is not a bug in your code or a missing compiler flag; it is a single ratio, FLOPs per byte moved, and almost every perform…