Hardware-Aware Architecture Search
Why optimising a model for FLOPs produces slow models, how measured latency became the objective instead, and what makes once-for-all supernet training the practical form of the idea.
Early neural architecture search minimised FLOPs subject to an accuracy target and produced architectures that were slower on real devices than the hand-designed baselines they beat on paper. The diagnosis was straightforward once stated: FLOPs are a poor proxy for time. Fixing the objective, rather than the search algorithm, is what made the field useful.
Why FLOPs mislead
Two operations with identical FLOP counts can differ by an order of magnitude in runtime. Depthwise separable convolutions have very low arithmetic intensity, so they are memory-bound and utilise a fraction of peak throughput while their FLOP count suggests they should be nearly free. Grouped convolutions fragment memory access. Small channel counts leave tensor cores underutilised. Kernel launch overhead is per-operation and invisible to a FLOP count, so a network of many tiny layers pays a cost no analytical model captures.
The consequence is that the ranking of architectures by FLOPs and by measured latency differ substantially, and they differ differently on different hardware. A model tuned for a mobile CPU is not the model you would choose for a mobile GPU or an NPU.
Measuring instead of modelling
The fix is to put real latency in the objective. Since measuring every candidate on device would be far too slow, the standard approach builds a latency lookup table: benchmark each operator type at each configuration once, then estimate a candidate architecture's latency by summing its operators' measured costs.
This is an approximation, and its main error is ignoring fusion, since a fused sequence is faster than the sum of its parts. It is nonetheless close enough to rank candidates correctly, which is all the search needs, and it turns an on-device measurement problem into a table lookup.
MnasNet and the hardware-aware family that followed optimise a joint objective, typically accuracy multiplied by a latency penalty term, tuned so that the search trades a fixed amount of accuracy per millisecond rather than treating latency as a hard constraint.
Once-for-all training
Searching architectures by training each candidate is prohibitive. The technique that made this practical is to train one over-parameterised supernet containing all candidates as subnetworks, with weight sharing, using progressive shrinking so that subnetworks of every size are trained to be individually usable (Cai et al., 2020, Once-for-All, arXiv:1908.09791).
After one training run, extracting a specialised architecture for a new device is a search over the supernet using the latency table, with no retraining. That is the property that matters commercially: one expensive training run amortises across every deployment target, which is what turns architecture search from a research project into an engineering process.
When it breaks
The latency table is a snapshot. It is valid for one device, one driver version, one runtime and one set of compiler flags. A framework update that improves a fused kernel changes the true ranking and the table does not know. Tables need re-benchmarking on a schedule, and searches run against stale tables optimise for a machine that no longer exists.
Weight sharing biases the ranking. Subnetworks in a supernet interfere with each other during training, so a subnetwork's accuracy inside the supernet is not its accuracy when trained standalone, and the correlation between the two is imperfect. Progressive shrinking and careful sampling reduce the gap without closing it, so the search's top candidate is often not the true optimum.
Searched architectures are fragile to task change. An architecture found for ImageNet classification at a given latency does not transfer cleanly to detection, segmentation, or a different input resolution. The search optimised a specific accuracy-latency point on a specific task, and the result should be treated as tuned to it.
The search space contains the answer or it does not. NAS explores a space a human defined, so it finds the best combination of the components it was given and cannot invent an operator. The genuinely large architectural changes of the last decade came from human design, and search has been a tool for tuning within an established family rather than for discovering new ones.
12 flashcards for this concept
Click a card to reveal the answer.