Compilation Targets and Runtime Fragmentation
The path from a trained PyTorch model to something that runs on a phone, why the intermediate format is where most deployment failures happen, and what each of the major runtimes assumes.
A trained model is a Python object holding tensors and a graph implied by control flow. A deployed model is a serialised artefact executed by a runtime with no Python. The conversion between them is where most on-device projects lose their time, and understanding what the conversion assumes is what avoids that.
The general shape
Every path follows the same three stages. Capture turns the eager model into an explicit graph, historically by tracing, increasingly by symbolic export. Optimisation rewrites that graph: constant folding, operator fusion, layout transformation, quantisation, and dead code removal. Lowering emits a serialised artefact for a specific runtime and target.
Capture is where models break, because a traced graph records only the path taken by the example input. Data-dependent control flow, Python-side loops over dynamic lengths, and shape-dependent branches all silently bake in one behaviour. A model that works correctly for the traced sequence length and produces wrong results for another is the classic symptom, and it is a correctness bug rather than a performance one.
The runtimes and what they assume
Core ML targets Apple platforms exclusively and is the only path to the Neural Engine. Conversion is from PyTorch or ONNX, the operator coverage is good, and the compiler decides op placement across CPU, GPU and Neural Engine with limited developer control. That last property is the main frustration: placement decisions are opaque and change between OS versions.
LiteRT, the successor name to TensorFlow Lite, is the mainstream Android path and reaches vendor NPUs through delegates. Delegate support varies by vendor and by device, so the same artefact accelerates on one phone and falls back on another.
ONNX Runtime is the portable option, with execution providers for many backends. Portability is genuine and the tradeoff is that it rarely extracts as much performance as a vendor-native path.
ExecuTorch is the PyTorch-native edge runtime, built on the newer export mechanism and designed to keep the graph closer to what the author wrote, with a smaller runtime and partitioner-based backend delegation.
GGUF with llama.cpp occupies a different niche: a single-file quantised format with a self-contained C++ runtime, dominant for local LLM inference across desktop and mobile precisely because it avoids the whole conversion ecosystem.
When it breaks
Numerical differences appear at conversion. Fused operations reassociate floating-point arithmetic, so converted models produce slightly different outputs. Usually harmless, occasionally not, particularly where a threshold or an argmax sits near a boundary. Validating a converted model against the original on real inputs, comparing distributions rather than checking a single output, belongs in the build.
Quantisation happens in the converter, using its own calibration. The converter's quantisation defaults may differ from what was validated in training, and per-tensor versus per-channel choices are often made silently. A model that passed evaluation in the training framework can fail after conversion for this reason alone.
Operator coverage is the real constraint on architecture. A novel attention variant or a custom activation may have no equivalent in the target runtime, forcing either a rewrite or a slow fallback. This is why edge model design is conservative, and why checking operator support belongs at the start of a project rather than at the end.
Every runtime version is a new compatibility surface. Artefacts are tied to runtime versions, and an app supporting several OS versions may need several artefacts. Build pipelines that produce and validate one model per target multiply quickly, and the matrix is the hidden cost of on-device deployment.
12 flashcards for this concept
Click a card to reveal the answer.