Running Language Models on a Phone: Memory Bandwidth, NPUs, and the Few-Billion-Parameter Ceiling

In April 2024 Microsoft reported that Phi-3-mini, 3.8 billion parameters quantised to 4 bits so that it occupied about 1.8 GB, generated "over 12 tokens per second" natively on an iPhone with an A16 Bionic chip (Abdin et al., 2024, arXiv:2404.14219). Two months later Apple said its roughly 3-billion-parameter on-device model, compressed to an average of 3.7 bits per weight, generated 30 tokens per second on an iPhone 15 Pro (Apple, 2024). Multiply model size by speed and an odd pair of numbers falls out: about 22 GB per second for the first phone, about 42 GB per second for the second. Neither is a compute figure. Both are, to a first approximation, the rate at which the phone's memory handed bytes to its processor.

A token of generation needs a few billion arithmetic operations, which a phone has to spare, and every weight read once, which it does not. The result is a calculable ceiling of a few billion dense parameters at 3 to 4 bits, and the shipping products (Gemini Nano at 1.8B and 3.25B, Apple's ~3B, Phi-3-mini, Llama 3.2 1B and 3B) cluster exactly there.

Why this matters: On-device features are routinely sized by NPU TOPS and parameter count, the wrong axis. Reply speed is bytes per token over effective memory bandwidth; prompt latency is compute; heat and battery decide how long either lasts.

TL;DR

• Decode is bandwidth-bound. A Snapdragon 8 Gen 3 bus is 68 GB/s in theory and 40 to 45 GB/s when one processor drives decode (Chen et al., SOSP 2025).
• A 3.2B model at 4.5 effective bits reads about 2 GB per token with its KV cache: an estimated 20 tokens/s on that SoC, about 15 once warm.
• Prefill is compute-bound. A 1,000-token prompt through a 3B model is about 6.4 trillion FLOPs, over six seconds on a phone GPU, which is why NPU offload reports 22x prefill gains (Xu et al., ASPLOS 2025).
• The ceiling is two walls meeting. At 42 GB/s, 15 tokens/s and 4.5 bits the bandwidth wall is about 5B parameters; a 3 GB memory budget puts the capacity wall in the same place. Two-bit weights move both, at a cost: Apple's model fell from 67.8 to 64.4 MMLU (Apple, 2025).
• NPUs are not uniformly faster. A 2026 study found CPUs up to 1.6x faster at prefill, NPU decode gains of 1.05 to 1.2x, and up to 51% more energy with heavier offload (Li et al., 2026).
• Heat and battery set duration. An iPhone 14 Pro reached 47.9°C after one conversation with a 4-bit 3B model (Laskaridis et al., MobiCom 2024).

At a Glance

flowchart LR
    P["Prompt tokens"] --> PF["Prefill: compute-bound"]
    PF --> KV["KV cache in LPDDR"]
    KV --> DL["Decode loop: bandwidth-bound"]
    DL --> KV
    DL --> OUT["Streamed tokens"]
    NPU["NPU, high FLOPS"] -.-> PF
    GC["GPU or CPU, flexible shapes"] -.-> DL
    MEM["LPDDR5X, about 40-68 GB/s"] -.-> DL
    TH["Thermal and battery budget"] -.-> DL
    TH -.-> PF

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    class P blue
    class PF,DL purple
    class KV,GC,NPU slate
    class OUT teal
    class MEM amber
    class TH rose

Prefill processes the prompt in large matrix multiplications and is limited by arithmetic, so the NPU helps. Decode rereads every weight and the growing KV cache for each token and is limited by the memory bus.

From Pocket Assistants to Pocket Transformers

Phones have run neural networks for a decade, but keyboard and photo models were megabytes and compute-bound, so the first mobile NPUs were built as fixed-function INT8 matrix engines. A language model is gigabytes read in full per token, and the constraint moved to the memory bus. The tool for that shift predates it: in 2009 Williams, Waterman and Patterson published the Roofline model, which bounds a kernel's performance by the lesser of peak compute and memory bandwidth times the kernel's arithmetic intensity, its operations per byte moved (Williams, Waterman & Patterson, 2009). Batch-one decode sits deep in the bandwidth-limited slope.

Open weights brought the software quickly. Georgi Gerganov published llama.cpp on 10 March 2023, putting LLaMA on ordinary CPUs. In May 2023 MLC LLM ran a 4-bit Vicuna-7B on a Galaxy S23's Adreno GPU using TVM-generated OpenCL kernels (MLC LLM, 2023). In December 2023 Google shipped Gemini Nano on the Pixel 8 Pro: two variants of 1.8B and 3.25B parameters, distilled from larger Gemini models and "4-bit quantized for deployment" (Gemini Team, 2023). The same month Apple posted LLM in a Flash, treating DRAM as a cache over flash storage (Alizadeh et al., 2023).

timeline
    title On-device language models, from theory to shipping products
    2009 : Roofline model bounds performance by compute and bandwidth
    2023 : llama.cpp runs LLaMA on commodity CPUs
         : MLC LLM runs 4-bit Vicuna-7B on a phone GPU
         : Gemini Nano ships on Pixel 8 Pro
         : LLM in a Flash runs models twice the size of DRAM
    2024 : Phi-3-mini reports over 12 tokens per second on an A16
         : Apple reports about 3B at 3.7 bits and 30 tokens per second
         : PowerInfer-2 and quantised Llama 3.2 1B and 3B
    2025 : Apple moves its on-device model to 2-bit QAT
         : ExecuTorch 1.0 and HeteroInfer GPU-NPU decode
    2026 : Stage-level studies find NPUs not always faster

2024 was the year of measurement. MELT benchmarked llama.cpp and MLC LLM across phones and Jetson boards and concluded that "LLM inference is largely memory-bound" (Laskaridis et al., 2024). Meta's quantised Llama 3.2 used 4-bit groupwise weights (group size 32) with 8-bit dynamic activations, reporting 2 to 4x speedups on a OnePlus 12 (Meta, 2024). In 2025 ExecuTorch 1.0 promoted its Core ML, XNNPACK and Qualcomm Hexagon backends to production (PyTorch, 2025).

How On-Device Inference Actually Works

Two phases, two bottlenecks

A transformer with \(N\) parameters spends roughly \(2N\) operations per token. Prefill does this for all \(L_p\) prompt tokens at once; decode for one token, reusing cached keys and values. Yet every forward pass streams the full weights through the processor once, however many tokens it carries.

Let \(b\) be effective bits per weight including quantisation scales, so weights occupy \(W = Nb/8\) bytes. The arithmetic intensity of one decode step is

four FLOPs per byte at 4 bits. For a prefill of \(L_p\) tokens the weights are still read once while compute grows with the prompt:

A 1,000-token prompt at 4 bits has intensity 4,000, three orders of magnitude to the right of decode. The Roofline bound is attainable FLOPS \(= \min(P, \; I \cdot B)\), with \(P\) the processor's peak FLOPS and \(B\) its effective bandwidth in bytes per second. With \(B \approx 4 \times 10^{10}\) and \(P \approx 10^{12}\), decode reaches \(I \cdot B = 1.6 \times 10^{11}\) FLOPS, a sixth of what the processor could do, while prefill saturates \(P\) long before bandwidth matters. The same model on the same phone is memory-starved when it writes and compute-starved when it reads.

The decode formula, and what it hides

Dividing bandwidth-limited FLOPS by FLOPs per token gives

where \(W\) is weight bytes touched per token and \(K(c)\) is KV cache bytes read at context length \(c\) with \(b_{\text{kv}}\) bits per cached value. Three parts are routinely misread.

\(B_{\text{eff}}\) is not the datasheet figure. An LPDDR5X x64 package with two devices advertises 68 GB/s aggregate, and LPDDR5T at 9.6 Gbps 76.8 GB/s (Kamath, 2024). Kamath's example of a 3.5 GB INT4 LLaMA 2 7B at 19 tokens/s is exactly \(68/3.5\), the formula at its theoretical best. Chen et al. measured a Snapdragon 8 Gen 3 and found 61.9 GB/s achievable, only 40 to 45 GB/s when a single processor drives decode, and about 60 GB/s when GPU and NPU read concurrently (Chen et al., 2025).

\(W\) is bytes touched, not model size: tied embeddings count because the output projection reads them every step, while sparse models touch less.

\(K(c)\) grows with context and is reread every token. Apple's 2025 on-device model splits its depth into two blocks of 62.5% and 37.5%, with the second reusing the first block's keys and values. That removes 37.5% of KV memory and about 37.5% of time to first token, and the cache is stored at 8 bits (Apple, 2025).

[IMAGE: Roofline plot for a Snapdragon 8 Gen 3 class SoC, log-log axes, x = arithmetic intensity in FLOPs per byte, y = attainable FLOPS. A bandwidth slope at about 42 GB/s meets two flat compute ceilings, GPU near 1 TFLOPS and NPU near 10 TFLOPS. Decode of a 4-bit model is a point at intensity 4 on the slope; prefill of a 1,000-token prompt at intensity 4,000 sits under the compute ceilings. Caption: "Same model, same phone, two regimes: decode lives on the bandwidth slope, prefill under the compute roof."]

The capacity wall, and where the walls meet

MobileLLM argues a mobile app "should not exceed 10% of the DRAM" (Liu et al., 2024). System-owned models escape that rule but still compete with foreground apps, and mobile operating systems kill processes rather than swap. If a model may hold \(M\) bytes, \(N \le 8M/b\). Adding a minimum acceptable decode speed \(T_{\min}\) and ignoring the KV term:

With \(B_{\text{eff}} = 42\) GB/s, \(T_{\min} = 15\) and \(b = 4.5\), the bandwidth term is \(8 \times 42/(15 \times 4.5) \approx 5.0\) billion parameters. With \(M = 3\) GB the capacity term is \(8 \times 3/4.5 \approx 5.3\) billion. Both walls sit near five billion and scale with \(1/b\), so bits are the lever everyone pulls. Apple's 2025 model uses 2 bits per weight via quantisation-aware training with learnable weight clipping, a 4-bit embedding table, and LoRA recovery adapters. The price, in Apple's own table: MMLU drops from 67.8 to 64.4 and instruction following from 85.1 to 82.3.

Energy follows bytes moved

Decode energy tracks bytes moved as speed does. MobileLLM's rule of thumb is about 0.1 J per token per billion parameters, so a 7B model at 10 tokens/s drains a roughly 50 kJ iPhone battery in under two hours (Liu et al., 2024). MELT measured an iPhone 14 Pro sustaining up to 13.8 W during inference, far more than a phone dissipates indefinitely; thermal management lowers operating frequencies, and \(B_{\text{eff}}\) falls mid-answer.

What the NPU is actually for

An NPU earns its FLOPS per watt by committing to a narrow execution model: static graphs, fixed tensor shapes, and quantisation it can implement in fixed-function blocks, typically per-tensor INT8. Chen et al. report a phone GPU achieving about 1 TFLOPS in practice against a theoretical 2.8, while the NPU delivered up to 10 TFLOPS when shapes aligned. That is decisive for prefill and nearly irrelevant for decode, where the processor waits on memory anyway.

llm.npu attacks three mismatches (Xu et al., 2025): variable-length prompts are cut into fixed-size chunks for static graphs; activation outliers that per-tensor INT8 cannot represent run on the CPU or GPU in parallel; and quantisation-sensitive blocks are scheduled out of order to other processors. The payoff is over 1,000 prefill tokens/s for a billion-parameter model, 22.4x faster prefill and 30.7x less energy than its baselines on average. Decode still ran on the CPU.

Seeing It in Motion

Inside the SoC

flowchart TB
    subgraph SOC["Phone SoC, unified memory"]
        direction TB
        RT["Runtime and graph partitioner"]
        CPU["CPU cores: every operator"]
        GPU["GPU: dynamic shapes"]
        NPU["NPU: static INT8 graphs"]
    end
    FL["UFS flash: weights on disk"] --> MM["mmap page cache"]
    MM --> LP["LPDDR5X DRAM"]
    LP --> CPU
    LP --> GPU
    LP --> NPU
    RT --> CPU
    RT --> GPU
    RT --> NPU
    NPU -.->|"unsupported op"| FB["Fallback partition"]
    FB -.-> CPU

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    class FL blue
    class MM,RT slate
    class LP amber
    class CPU,GPU,NPU purple
    class FB rose

Unified memory does not mean no copies: NPU buffers are often not laid out as GPU kernels expect, so each partition boundary can cost a synchronisation and a transfer.

[IMAGE: Exploded phone logic board: SoC with CPU, GPU and NPU, LPDDR5X stacked on top, UFS 4.0 flash beside it, arrows labelled about 40-68 GB/s and up to 4.2 GB/s sequential. Caption: "Weights travel from flash once at load, and across the memory bus for every token."]

One request across processors

sequenceDiagram
    participant App
    participant RT as Runtime
    participant NPU
    participant GPU
    participant MEM as LPDDR memory
    App->>RT: Prompt of 1,000 tokens
    RT->>NPU: Fixed-size prefill chunks
    NPU->>MEM: Write keys and values
    RT->>GPU: Outlier channels in parallel
    NPU-->>RT: Prefill done
    RT-->>App: First token
    loop Every generated token
        GPU->>MEM: Read all weights and KV cache
        GPU-->>App: Next token
    end
    Note over GPU,MEM: Speed set by bytes per second, not FLOPS

The NPU consumes the prompt; a flexible processor owns the loop. HeteroInfer also decodes on GPU and NPU together because two concurrent readers pulled about 60 GB/s from a bus one processor used at 40 to 45 (Chen et al., 2025).

The thermal lifecycle

stateDiagram-v2
    [*] --> Cold
    Cold --> Loading: model mapped from flash
    Loading --> Peak: weights paged in
    Peak --> Warm: sustained decode
    Warm --> Throttled: skin temperature limit
    Throttled --> Warm: load drops
    Warm --> Peak: device cools
    Throttled --> Evicted: memory pressure
    Evicted --> Cold

Benchmarks report Peak; users live in Warm and Throttled. MELT saw throughput step down during continuous prompting on an iPhone 14 Pro, and model loads averaging 2.41 seconds (Laskaridis et al., 2024).

By the Numbers

Sources: Abdin et al. (2024); Apple (2024, 2025); Kamath (2024); Chen et al. (2025); Xu et al. (2025); Li, Qi & Chen (2026); Xue et al. (2024); Laskaridis et al. (2024), all linked inline above and listed below. "Implies" figures are our division of model bytes by tokens/s, ignoring KV reads and undisclosed details such as speculation; vendor figures are claims under their own test conditions.

The implied figures for a 2022 A16 and a 2023 A17 Pro bracket the 40 to 45 GB/s Chen et al. measured on a Snapdragon: a phone of that era moved a few tens of gigabytes of weights per second.

[IMAGE: Scatter plot, x = bytes read per token in GB, y = decode tokens/s, with points for Phi-3-mini, Apple's model, HeteroInfer Llama-8B and Kamath's 7B example over hyperbolas for 20, 40 and 68 GB/s. Caption: "Phone decode results fall on constant-bandwidth hyperbolas, not on any compute curve."]

A Concrete Example

Estimate decode speed for a 3B-class model at 4 bits on a Snapdragon 8 Gen 3 phone. Every number is cited or an explicit assumption; the result is an estimate, not a benchmark.

Step 1: count the weights. Take Llama 3.2 3B's published configuration: hidden size 3,072, 28 layers, 24 query heads, 8 KV heads of dimension 128, MLP width 8,192, vocabulary 128,256, tied embeddings. Attention per layer is \(2 \times 3072^2 + 2 \times 3072 \times 1024 = 25.17\)M parameters; the gated MLP is \(3 \times 3072 \times 8192 = 75.49\)M. Twenty-eight layers give 2,818.5M, the embedding matrix 394.0M, total about 3.21B.

Step 2: bytes touched per token. Assume 4-bit groupwise weights with group size 32 and a 16-bit scale per group, so \(b = 4 + 16/32 = 4.5\). Tied embeddings mean the output projection reads the full matrix every step, so \(W = 3.21 \times 10^9 \times 4.5/8 = 1.81\) GB.

Step 3: add the KV cache. Each context token stores \(2 \times 28 \times 8 \times 128 = 57{,}344\) values, 114.7 KB at 16 bits. Midway through a reply to a 1,000-token prompt, \(c = 1{,}500\) and \(K = 0.17\) GB. Bytes per step: 1.98 GB.

Step 4: divide by bandwidth.

Step 5: confirm compute is not the limit. FLOPs per token are about \(2 \times 3.21 \times 10^9\) plus roughly \(2 \times 28 \times 1500 \times 3072 = 2.6 \times 10^8\) for attention over the cache, about \(6.7 \times 10^9\) total. A GPU at 1 TFLOPS could supply about 150 tokens/s; bandwidth caps us near 21.

Step 6: cost the prompt. Prefill of 1,000 tokens is about \(6.4 \times 10^{12}\) FLOPs: 6.4 seconds at 1 TFLOPS. HeteroInfer's Llama-8B prefill of 247.9 tokens/s implies \(2 \times 8 \times 10^9 \times 248 \approx 4\) TFLOPS effective, which would prefill our model in about 1.6 seconds; an idealised NPU at 10 TFLOPS, about 0.64 seconds.

Step 7: apply heat and battery. Assume sustained decode loses 25 to 35% after a minute of load (an illustrative assumption, not a measured constant): about 13 to 17 tokens/s warm. At 0.1 J per token per billion parameters a 300-token reply costs about 96 J, so a 50 kJ battery covers roughly 520 replies; MELT estimated 490 to 590 prompts per charge on its devices. As a cross-check, HeteroInfer's Llama-8B at 14.01 tokens/s would imply about 63 GB/s if its weights were near 4.5 bits (our assumption), close to the 60 GB/s the paper reports for concurrent decode.

The estimate: about 20 tokens/s cold, about 15 warm, 1.6 seconds to first token. At 8B, single-processor decode falls below 10; at 2 bits it climbs toward 35, if quality survives.

[IMAGE: Waterfall chart: 34.3 tokens/s theoretical, 31.3 achievable, 21.5 single processor, 15 warm (estimate), with a side panel of 1,000-token prefill at 6.4 s, 1.6 s and 0.64 s. Caption: "Where the tokens per second go for a 3.2B model at 4.5 bits. Estimates, labelled."]

Where It Breaks

The NPU that makes things slower

The comfortable recipe, NPU for prefill and GPU for decode, is contested. Li, Qi and Chen decomposed a CPU-NPU pipeline stage by stage and found the CPU up to 1.6x faster at prefill, NPU decode gains of only 1.05 to 1.2x, and energy rising up to 51% with more offload, blaming scheduling overhead and cross-backend fallback (Li, Qi & Chen, 2026). Cai et al., profiling five frameworks including llama.cpp and Qualcomm's GENIE, found framework-induced gaps amplified up to 10x on NPUs when implementations used custom operators, and up to 40% energy wasted by uncoordinated NPU sleep latencies and CPU polling (Cai et al., 2026). Beside llm.npu's 22.4x, the disagreement is not about silicon but whether the stack hands the NPU large, fixed-shape, per-tensor-quantised work.

Operator coverage and static shapes

Rotary embeddings, attention over a growing cache and custom normalisations are frequent NPU casualties, each creating a fallback partition. Decode shapes change every step, and HeteroInfer reports NPU performance degrading at small sequence lengths. A model designed without the NPU's operator list can run mostly on the CPU with the NPU nominally enabled.

The KV cache eats the bandwidth

In the worked example the cache was 9% of bytes per step. At 16,000 tokens and 16 bits it is 1.84 GB, more than the weights, so decode roughly halves while capacity shrinks. KV quantisation and sharing become prerequisites, with quality costs short-context benchmarks hide.

The quantisation cliff

MELT found some models of 3B parameters or fewer "unusable with 3-bit precision" under post-training quantisation, and its 4-bit variants also decoded faster than 3-bit ones (Laskaridis et al., 2024). Apple reached 2 bits only with QAT plus recovery adapters and still gave up 3.4 MMLU points. Small models have the least redundancy to absorb quantisation error, so the regime that most needs low bits tolerates them worst.

Memory pressure and cold starts

Memory-mapped weights let the OS reclaim pages instead of killing the process, but reclaimed pages fault back from flash on the next request. UFS 4.0 reads up to 4,200 MB/s sequentially (Samsung, 2022), yet page faults are scattered reads, so occasionally used features pay seconds of warm-up on nearly every use.

Flash offload without sparsity

LLM in a Flash works because FFN activations in ReLU-style models are sparse and predictable: windowing reuses recently active neurons, and row-column bundling turns scattered reads into contiguous chunks (Alizadeh et al., 2023). PowerInfer-2 schedules neuron clusters, dense ones on the NPU and sparse ones on the CPU, to serve a ReLU-sparsified TurboSparse-Mixtral-47B at 11.68 tokens/s (Xue et al., 2024). Both need exploitable activation sparsity; with dense activations, every gigabyte read from flash costs roughly ten times one in DRAM.

[IMAGE: Line chart of decode tokens/s over 10 minutes of continuous generation for a 3B 4-bit model, with device temperature on a second axis; throughput steps down at DVFS transitions as temperature climbs toward 48°C, and a shaded band marks where a 20-token benchmark stops. Caption: "The benchmark ends where the user's experience begins. Illustrative shape based on MELT's trends."]

Alternative Designs

Each design answers a different binding constraint. For app developers who cannot retrain, the honest default is a dense 1B to 4B model at 4 bits on GPU or CPU, with harder requests sent to a server.

How It Is Used in Practice

Apple runs the most integrated stack: grouped-query attention, palettisation averaging 3.7 bits and token speculation in 2024 (Apple, 2024), then 2-bit QAT with shared 8-bit KV in 2025, opened to third-party apps (Apple, 2025). Google ships Gemini Nano as a system service behind Android AICore, so one resident copy serves many features; its two sizes target "low and high memory devices respectively" (Gemini Team, 2023), the capacity wall written into a product line.

ExecuTorch powers on-device features across Instagram, WhatsApp, Messenger and Facebook (PyTorch, 2025), and Meta co-developed quantised Llama 3.2 with Qualcomm and MediaTek so the scheme matched SoC kernels (Meta, 2024). llama.cpp and MLC LLM remain the portable default, trading NPU access for one codebase.

Teams that succeed measure on the lowest-tier supported device: time to first token at the real prompt length, decode after several minutes, peak memory, and energy per task. They cache system-prompt KV state, scope features to what a 3B model does reliably (summarisation, rewriting, extraction), and build an escalation path to a server.

[IMAGE: Decision flowchart from prompt length, reply length, quality and device tier to "sub-billion always-on", "3B at 4 bits on GPU", "NPU prefill plus GPU decode", "sparse flash offload" or "escalate to server". Caption: "Choose from the binding constraint, not the TOPS rating."]

Insights Worth Remembering

1. Model bytes times tokens per second is a bandwidth measurement. If a vendor's implied bandwidth exceeds the memory spec, something else is at work: speculation, sparsity, or fewer bytes than advertised.

2. TOPS predicts prefill, not reply speed. At four FLOPs per byte the processor idles most of each decode step. Buying FLOPS for decode is buying a faster car for a traffic jam.

3. The few-billion ceiling is two walls meeting. Bandwidth and app memory both land near 5B parameters at 4.5 bits, and both move with \(1/b\).

4. Bits are cheaper for a platform owner than for an app developer. Apple's 2 bits needed QAT, learnable clipping and recovery adapters. Without training access, treat 4 bits as the floor for small models.

5. Heterogeneous decode is a bandwidth trick. HeteroInfer's GPU plus NPU decode helps because two readers extract about 60 GB/s where one gets 40 to 45, not because of added FLOPS.

6. Whether the NPU helps is a software property. Similar silicon yields 22x faster prefill in one system and slower, hungrier prefill in another. Partition count and quantisation alignment decide which result you reproduce.

7. Sparsity is the only route past the ceiling that keeps the bits. Flash offload reaches 47B on a phone because most weights are never read for a given token, and that sparsity must come from the model architecture.

Open Questions

Will bandwidth grow fast enough to move the ceiling? Measured: LPDDR5T reaches 76.8 GB/s against LPDDR5X's 68, a modest step. Unknown: whether wider buses or processing-in-memory fit phone prices and power.

Is 2 bits a floor or a waypoint? Shown: Apple's 2-bit QAT costs a few MMLU points at 3B. Not shown: whether that holds for reasoning or long-context tasks, where the evidence is still mostly vendor-reported.

Can NPUs become good decode engines? Measured: at most 1.2x decode gains in one 2026 study. Speculative: dynamic shapes and native per-group quantisation would help, but a bandwidth-bound phase cannot outrun its memory bus.

Does speculative decoding change the rule on phones? Known: verifying several tokens per weight read raises arithmetic intensity. Unknown: sustained gains once draft memory, real acceptance rates and heat are counted.

How should on-device inference be benchmarked? MELT and the 2026 studies show large variance from run length, DVFS state and framework, with no agreed protocol for warm decode and energy per task. Cross-paper speedups are indicative, not comparable.

Sources and Further Reading

1. Williams, S., Waterman, A., & Patterson, D. (2009). "Roofline: An Insightful Visual Performance Model for Multicore Architectures." Communications of the ACM, 52(4). doi:10.1145/1498765.1498785
2. Alizadeh, K., et al. (2023). "LLM in a flash: Efficient Large Language Model Inference with Limited Memory." ACL 2024. arXiv:2312.11514
3. Xu, D., et al. (2024). "Fast On-device LLM Inference with NPUs." ASPLOS 2025. arXiv:2407.05858
4. Chen, L., et al. (2025). "Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference." SOSP 2025. arXiv:2501.14794
5. Xue, Z., et al. (2024). "PowerInfer-2: Fast Large Language Model Inference on a Smartphone." arXiv:2406.06282
6. Laskaridis, S., Katevas, K., Minto, L., & Haddadi, H. (2024). "MELTing Point: Mobile Evaluation of Language Transformers." MobiCom 2024. arXiv:2403.12844
7. Li, P., Qi, J., & Chen, Q. (2026). "When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference." arXiv:2605.27435
8. Cai, G., et al. (2026). "Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference." arXiv:2607.05475
9. Apple (2025). "Apple Intelligence Foundation Language Models: Tech Report 2025." arXiv:2507.13575
10. Apple Machine Learning Research (2024). "Introducing Apple's On-Device and Server Foundation Models." machinelearning.apple.com
11. Abdin, M., et al. (2024). "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone." arXiv:2404.14219
12. Gemini Team, Google (2023). "Gemini: A Family of Highly Capable Multimodal Models." arXiv:2312.11805
13. Liu, Z., et al. (2024). "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases." ICML 2024. arXiv:2402.14905
14. Meta AI (2024). "Introducing quantized Llama models with increased speed and a reduced memory footprint." ai.meta.com
15. PyTorch (2025). "Introducing ExecuTorch 1.0: Powering the next generation of edge AI." pytorch.org
16. MLC LLM (2023). "Bringing Hardware Accelerated Language Models to Android Devices." blog.mlc.ai
17. Kamath, N. (2024). "LPDDR Memory Is Key For On-Device AI Performance." Semiconductor Engineering. semiengineering.com
18. Samsung Semiconductor (2022). "Samsung Develops First UFS 4.0 Storage Solution Compliant with New Industry Standard." news.samsungsemiconductor.com

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.