Mobile NPUs and the Accelerator Zoo

A modern phone contains at least three processors that can run a neural network, and they are not interchangeable. Choosing the wrong one, or more commonly having the runtime choose for you, produces order-of-magnitude differences in latency and battery drain for identical model weights.

The three targets

CPU is universally available, supports every operator, and handles dynamic shapes and control flow without complaint. It is also the slowest and the least power-efficient per operation, and it competes with the application for cores. It is the correct target for small models, for preprocessing, and as the honest baseline everything else is measured against.

GPU offers good throughput on large, regular, parallel work and is well supported through Metal on iOS and OpenCL or Vulkan on Android. Its weakness on mobile is that kernel launch overhead and CPU-GPU transfer costs are significant relative to small workloads, so a model with many tiny operations can be slower on the GPU than on the CPU. It also contends directly with UI rendering, which is visible to the user in a way that CPU contention is not.

NPU, marketed variously as a neural engine, tensor processor or AI accelerator, is a fixed-function matrix engine. It offers the best performance per watt by a wide margin, often several times the GPU's, and it achieves that by supporting a narrow set of operations at specific precisions, usually INT8 with limited floating-point support. It cannot execute arbitrary graphs.

The fallback problem

The consequence of that narrowness is the single most important practical fact about NPUs: an unsupported operator anywhere in the graph causes a fallback, and fallbacks are expensive.

A runtime that meets an unsupported op typically partitions the graph, running the supported segments on the NPU and the rest elsewhere. Each partition boundary costs a synchronisation and, often, a memory transfer between differently laid-out buffers. A model with three unsupported operators scattered through it can end up with seven partitions, and the transfer overhead exceeds everything the accelerator saved. The measured result is a model that runs slower with NPU acceleration enabled than without it, which is a genuinely common outcome and confusing until the partition count is inspected.

The practical discipline is therefore to design within the supported operator set from the start, verify partition counts as part of the build, and treat any new operator as a compatibility question before it is a modelling question.

When it breaks

Vendor variation is severe. Qualcomm's Hexagon, Apple's Neural Engine, Google's Tensor TPU and MediaTek's APU all support different operator sets, different quantisation schemes and different memory limits, and they are exposed through different APIs. A model validated on one Android flagship can fall back entirely on another. Cross-device testing is not optional, and the tail of devices is where the failures live.

Thermal throttling makes benchmarks lie. A phone sustains peak accelerator performance for seconds, not minutes. A benchmark that runs 100 inferences reports a number the device cannot sustain, and a continuously running feature such as live video processing operates in the throttled regime. Sustained-throughput measurement over several minutes is the number that predicts user experience.

Memory limits are hard and low. Mobile operating systems terminate applications that exceed a memory budget, often 1 to 3 GB depending on device and OS, and the model competes with the rest of the app. Weights must be memory-mapped rather than loaded, and activation peaks matter as much as weight size.

INT8 is often not optional. Where an NPU supports only INT8, a model that cannot be quantised to INT8 without unacceptable loss simply cannot use that accelerator. This makes quantisation robustness an architecture requirement at design time rather than a post-training step, which reverses the usual order of operations.