On-Device & Edge AI
Mobile NPUs, memory-bound inference on consumer silicon, compilation targets and privacy-driven local models.
7concepts
74flashcards
51minutes of reading
- 01 Federated Learning on Edge Devices Training a shared model without collecting the data, why non-IID client distributions break the averaging assumption, and the systems constraints that decide what is actually trainable.
- 02 On-Device LLM Inference Constraints The arithmetic that decides whether a language model can run on a phone, why bandwidth rather than compute is the binding constraint, and what the KV cache does to the memory budget.
- 03 TinyML and Microcontroller Inference On a microcontroller the binding constraint is not compute but two memories, flash for weights and code and SRAM for activations, and the techniques that work (static arenas, peak-memory-aware search, patch-based execution) all attack peak SRAM.