Depthwise Separable Convolutions and Mobile Networks

EfficientNet-B0 reaches 77.1 percent ImageNet top-1 with 0.39 billion FLOPs. ResNet-50 reaches 76.0 percent with 4.1 billion (Tan & Le, 2019, EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, ICML, arXiv:1905.11946). Better accuracy for a tenth of the arithmetic sounds like a tenfold speedup. On a datacenter GPU it often is not, and the reason lies in the building block both of Google's mobile network families share.

Factoring the convolution

A standard convolution with a \(D_K \times D_K\) kernel maps \(M\) input channels to \(N\) output channels over a \(D_F \times D_F\) feature map. Every output channel looks at every input channel at every spatial offset, so its multiply-add count is

A depthwise separable convolution splits this into two stages. A depthwise convolution applies one \(D_K \times D_K\) filter per input channel, doing spatial filtering with no channel mixing. A pointwise \(1 \times 1\) convolution then mixes channels with no spatial extent. The cost becomes

With a \(3 \times 3\) kernel and \(N = 256\), the ratio is \(1/256 + 1/9 \approx 0.115\), about 8.7 times cheaper, and parameters shrink by the same factor. The assumption baked in is that spatial correlation and cross-channel correlation can be learned separately with little loss.

MobileNet v1 stacked these blocks and added two knobs: a width multiplier \(\alpha\) that thins every layer, reducing cost roughly by \(\alpha^2\), and a resolution multiplier that cuts it by the square of the input scaling. The base model scored 70.6 percent top-1 with 569 million multiply-adds and 4.2 million parameters (Howard et al., 2017, MobileNets, arXiv:1704.04861).

Inverted residuals and linear bottlenecks

A classic ResNet bottleneck is wide, narrow, wide: it squeezes channels, convolves, and expands, with the skip connection joining the wide ends. MobileNet v2 inverts this (Sandler et al., 2018, MobileNetV2: Inverted Residuals and Linear Bottlenecks, arXiv:1801.04381). A block takes a thin tensor with \(k\) channels, expands it with a \(1 \times 1\) convolution to \(tk\) channels (typically \(t = 6\)), runs a cheap \(3 \times 3\) depthwise filter in that wide space, and projects back to a thin tensor. The skip connection joins the thin ends.

Two design choices carry the argument. The depthwise stage is cheap enough that doing it in the expanded space costs little, so the network filters in a rich representation. The final projection has no nonlinearity: the authors argue that ReLU applied in a low-dimensional space destroys information that cannot be recovered, so the bottleneck stays linear. The result was 72.0 percent top-1 at 300 million multiply-adds and 3.4 million parameters, better and cheaper than v1.

Compound scaling

Given a good small network, how should you spend a larger budget? Scaling only depth, only width, or only resolution saturates. EfficientNet scales all three together with one coefficient \(\phi\):

Because FLOPs grow linearly in depth and quadratically in width and resolution, the constraint makes total FLOPs grow by about \(2^\phi\). A small grid search on the baseline found \(\alpha = 1.2\), \(\beta = 1.1\), \(\gamma = 1.15\). Scaling up the NAS-found B0, built from inverted-residual blocks, gave B7 at 84.3 percent top-1, which the authors reported as 8.4 times smaller and 6.1 times faster at inference than the best previous ConvNet. For the general question of how to trade layers against channels, see depth versus width tradeoffs.

The FLOPs-latency gap, and the disagreement it caused

Multiply-adds count arithmetic, not memory traffic. A depthwise convolution does very little arithmetic per byte of activation it reads, so on hardware whose bottleneck is memory bandwidth and kernel parallelism it cannot keep the processor busy. ShuffleNet V2 made the general case that architecture comparisons should use the direct metric, measured speed on the target platform, because memory access cost and platform characteristics break the FLOPs proxy (Ma et al., 2018, ShuffleNet V2, arXiv:1807.11164). Hardware-aware architecture search builds that lesson into the search objective.

The sharpest evidence comes from the same authors as EfficientNet. EfficientNetV2 identified extensive depthwise convolutions as its predecessor's training bottleneck, noting they "often cannot fully utilize modern accelerators," and replaced the early stages with Fused-MBConv, which swaps the expansion and depthwise pair for one regular \(3 \times 3\) convolution. The same paper found that equally scaling every stage, the premise of compound scaling, is suboptimal (Tan & Le, 2021, EfficientNetV2, arXiv:2104.00298). The design that minimises FLOPs and the design that minimises GPU time are different networks, and which one counts as efficient depends on the hardware you name.

When it breaks

Quantization. The depthwise layers of MobileNetV2 have wildly different weight ranges per channel. Nagel et al. report a per-tensor INT8 quantization of the trained model falling from 71.72 percent to 0.12 percent top-1; per-channel scales or their weight-equalisation method recover about 70 to 71 percent (Nagel et al., 2019, Data-Free Quantization Through Weight Equalization and Bias Correction, arXiv:1906.04721). Deploying through an integer-only runtime without per-channel support is a known trap.

Activation memory. The \(6\times\) expansion means the widest tensors in the network live inside blocks, and early high-resolution stages dominate peak memory. That is small on a phone and decisive on a microcontroller.

Accelerator mismatch. On NPUs and GPUs whose kernels favour dense regular convolutions, a separable network can realise only a small part of its FLOP advantage over a ResNet, or none. Profile on the deployment target before accepting any efficiency claim stated in multiply-adds.