On-Device & Edge AI
Mobile NPUs, memory-bound inference on consumer silicon, compilation targets and privacy-driven local models.
5concepts
60flashcards
37minutes of reading
- 01 Compilation Targets and Runtime Fragmentation The path from a trained PyTorch model to something that runs on a phone, why the intermediate format is where most deployment failures happen, and what each of the major runtimes assumes.
- 02 Mobile NPUs and the Accelerator Zoo What a phone's neural processing unit is good at, why the same model runs at wildly different speeds on the CPU, GPU and NPU of one device, and the fallback that silently destroys performance.
- 03 Privacy-Driven Local Inference What running a model locally actually guarantees, which parts of the pipeline still leak, and how hybrid designs preserve most of the property while escalating the hard requests.
- 04 Federated Learning on Edge Devices Training a shared model without collecting the data, why non-IID client distributions break the averaging assumption, and the systems constraints that decide what is actually trainable.
- 05 On-Device LLM Inference Constraints The arithmetic that decides whether a language model can run on a phone, why bandwidth rather than compute is the binding constraint, and what the KV cache does to the memory budget.