Inference & Serving
24 min
Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling
Phones ship NPUs rated in trillions of operations per second, yet the speed at which a reply appears is set by how fast LPDDR memory can hand a couple of gigabytes of weights to the processor, over and over. This post derives the…