On-Device & Edge AI
Mobile NPUs, memory-bound inference on consumer silicon, compilation targets and privacy-driven local models.
7concepts
74flashcards
51minutes of reading
- 01 Cloud-Edge Split Inference Where to cut a model between device and datacenter is an optimisation over per-layer compute, activation size and link conditions; it works well for feed-forward vision models and poorly for autoregressive LLMs, which split at the query instead.
- 02 Compilation Targets and Runtime Fragmentation The path from a trained PyTorch model to something that runs on a phone, why the intermediate format is where most deployment failures happen, and what each of the major runtimes assumes.
- 03 Mobile NPUs and the Accelerator Zoo What a phone's neural processing unit is good at, why the same model runs at wildly different speeds on the CPU, GPU and NPU of one device, and the fallback that silently destroys performance.
- 04 Privacy-Driven Local Inference What running a model locally actually guarantees, which parts of the pipeline still leak, and how hybrid designs preserve most of the property while escalating the hard requests.
- 05 Federated Learning on Edge Devices Training a shared model without collecting the data, why non-IID client distributions break the averaging assumption, and the systems constraints that decide what is actually trainable.
- 06 On-Device LLM Inference Constraints The arithmetic that decides whether a language model can run on a phone, why bandwidth rather than compute is the binding constraint, and what the KV cache does to the memory budget.
- 07 TinyML and Microcontroller Inference On a microcontroller the binding constraint is not compute but two memories, flash for weights and code and SRAM for activations, and the techniques that work (static arenas, peak-memory-aware search, patch-based execution) all attack peak SRAM.