On-Device & Edge AI
Mobile NPUs, memory-bound inference on consumer silicon, compilation targets and privacy-driven local models.
5concepts
60flashcards
37minutes of reading
- 01 Compilation Targets and Runtime Fragmentation The path from a trained PyTorch model to something that runs on a phone, why the intermediate format is where most deployment failures happen, and what each of the major runtimes assumes.
- 02 Mobile NPUs and the Accelerator Zoo What a phone's neural processing unit is good at, why the same model runs at wildly different speeds on the CPU, GPU and NPU of one device, and the fallback that silently destroys performance.
- 03 Privacy-Driven Local Inference What running a model locally actually guarantees, which parts of the pipeline still leak, and how hybrid designs preserve most of the property while escalating the hard requests.