Inference & Serving
21 min
FlashAttention and the Memory Wall: Why Attention Was Never Compute-Bound
An A100 can execute 312 trillion half-precision operations per second but can only pull about 2 terabytes from memory in that same second. FlashAttention made attention fast not by computing less, but by refusing to touch memory.