TinyML and Microcontroller Inference

An STM32F746, a popular Arm Cortex-M7 part, has 320 kB of SRAM and 1 MB of flash. ResNet-50 exceeds that flash by roughly 100 times, and MobileNetV2, a network designed for phones, exceeds the SRAM by 22 times (Lin et al., 2020, MCUNet: Tiny Deep Learning on IoT Devices, NeurIPS, arXiv:2007.10319). There is no operating system to page memory out, no allocator you can trust over months of uptime, and often no floating-point budget. The on-device LLM constraints of a phone are three orders of magnitude away. What remains possible is a keyword spotter, a person detector, or a vibration anomaly model running for months on a small battery, and getting there means reasoning about memory before accuracy.

Two memories, two budgets

Flash is persistent and holds the program, the operator kernels and the quantized weights. SRAM is volatile and holds whatever the network computes at runtime. The two are budgeted separately, and they fail differently: exceed flash and the firmware does not link; exceed SRAM and it crashes at inference.

For a network executed layer by layer, the SRAM requirement is set by the worst single step, not the sum:

where \(\lvert a_\ell \rvert\) is the byte size of layer \(\ell\)'s output activation and \(M_{\text{runtime}}\) covers the interpreter's bookkeeping and scratch buffers. Both input and output must coexist while a layer runs. A worked case shows why early layers are the problem: a 224 by 224 input convolved to 112 by 112 with 32 channels at int8 produces \(112 \times 112 \times 32 = 401{,}408\) bytes, about 392 kB, which alone breaks a 320 kB budget before counting the input. Late layers with 7 by 7 maps are tiny by comparison.

The runtime: TensorFlow Lite Micro

TensorFlow Lite Micro takes the embedded constraints literally (David et al., 2020, TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems, arXiv:2010.08678). The application supplies one contiguous memory arena at start-up; the interpreter plans every tensor into it during initialisation and performs no allocation afterwards, which rules out heap fragmentation in long-running devices. An operator resolver links only the kernels the model uses, keeping flash small, and vendors can substitute optimised kernels such as Arm's CMSIS-NN without touching the model. On a SparkFun Edge board the paper reports a hotword model using 12.8 kB of total memory and a visual wake words model about 82 kB.

Models reach the device int8-quantized as a matter of course. The 4x flash saving over float32 matters, but so does the fact that many microcontrollers lack a floating-point unit or run floats far slower than integer arithmetic.

Designing the network for the memory

MCUNet argues that a runtime alone cannot close a 22x gap, and co-designs the network and the engine. TinyNAS first shrinks the search space, choosing input resolution and width multiplier so that sampled models are likely to fit the device's SRAM and flash, then searches within it. TinyEngine replaces interpretation with code generation and plans memory over the whole network rather than per layer. The system reached 70.7 percent ImageNet top-1 on an STM32H743 with 512 kB SRAM and 2 MB flash, using int4 weights, and reduced peak memory by 3.4 times against TFLM and CMSIS-NN.

MCUNetV2 then noticed that MobileNetV2's memory is wildly imbalanced: the first five blocks set the peak, the remaining thirteen are small (Lin et al., 2021, MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning, NeurIPS, arXiv:2110.15352). Running those early blocks on spatial patches, four by four, and stitching the outputs cut MobileNetV2's peak from 1,372 kB to 172 kB with unchanged accuracy. The price is repeated computation, because input patches must overlap to produce non-overlapping outputs; the authors shift receptive field to later stages to shrink that overlap. MCUNetV2 reported 71.8 percent ImageNet top-1 on a microcontroller and over 90 percent on visual wake words within 32 kB of SRAM.

Interpreter or compiler: an open disagreement

TFLM's authors argue that an interpreter, usually viewed as the slow option, suits embedded ML well, because runtime is dominated by a few heavy kernels and portability across a fragmented hardware ecosystem is worth more than the overhead. MCUNet's authors measured the opposite trade on their models: interpreter metadata consuming up to 65 percent of peak memory and adding 22 percent latency. Both are defensible. A vendor shipping one model to one chip should generate code; a platform supporting many models across many boards pays the interpreter's tax for maintainability.

When it breaks

Benchmarks on a desktop do not transfer. Accuracy measured in float on a workstation says little about the int8 or int4 model on the device, and latency and energy must be measured on silicon. MLPerf Tiny exists for this reason, measuring accuracy, latency and energy across keyword spotting, visual wake words, image classification and anomaly detection (Banbury et al., 2021, MLPerf Tiny Benchmark, arXiv:2106.07597).

Operator coverage is a hard wall. A layer the micro runtime does not implement cannot fall back to a CPU path the way it can on a phone. The architecture must be designed inside the supported operator set from the start.

Peak memory hides in the sensor pipeline. Audio feature extraction, image buffers from the camera and communication stacks share the same SRAM, and a model that fits in isolation can fail once integrated.

Updates are expensive. Replacing a model means a firmware or over-the-air flash write on a battery-powered device, so drift in the sensor or environment is corrected rarely, if ever.