On-Device LLM Inference Constraints

Whether a given language model runs usefully on a given phone is not a matter of opinion. It is three numbers: how many bytes the weights occupy, how fast the device moves bytes, and how much memory the operating system will let the application hold. Everything else is secondary.

The bandwidth calculation

Decoding is memory-bound. Every token requires reading all active weights from memory, so

A flagship phone in 2025 has roughly 50 to 70 GB/s of system memory bandwidth, an order of magnitude below a datacentre accelerator. A 3B-parameter model at 4 bits is about 1.5 GB, giving an upper bound near 40 tokens per second at 60 GB/s, and realistically 15 to 25 after accounting for achieved rather than peak bandwidth, the KV cache, and everything else the phone is doing.

That is enough for a responsive chat experience, since human reading speed is around 5 to 8 tokens per second. It is the reason the viable on-device size range settled at roughly 1B to 8B parameters at 3 to 4 bits: below that, quality suffers; above it, the model either does not fit in the memory budget or reads too slowly.

Prefill is a separate problem and often the worse one. It is compute-bound, and mobile compute is limited, so a 2,000-token prompt can take several seconds before the first token appears. This is why on-device assistants aggressively cache system prompts and keep contexts short, and why time to first token, not tokens per second, is usually the metric users complain about.

The memory budget

Three consumers compete. Weights are the largest and are fixed. The KV cache grows with context and concurrency. Activations peak during the forward pass and are transient but must fit.

The KV cache is the one that surprises people. For a model with \(L\) layers, \(H\) KV heads and head dimension \(d_h\), the cache is \(2 L H d_h\) values per token. A 3B model with 26 layers and 8 KV heads of dimension 128 stores about 53,000 values per token, or roughly 106 KB per token in fp16. At 8,000 tokens of context that is 850 MB, comfortably more than half the size of the quantised weights. Quantising the KV cache to INT8 or INT4 is therefore not an optimisation on mobile, it is a requirement for long context.

Weights must be memory-mapped from storage so the OS can evict pages under pressure rather than terminating the app. This makes the first token after a cold start slow, since the pages must be faulted in, and it makes storage read speed a component of user-visible latency.

When it breaks

Sustained generation throttles. Generating a long response holds the accelerator busy long enough for thermal limits to engage, so tokens per second measured over 20 tokens is not what a 500-token response delivers. The curve is not flat and the last part of a long answer can arrive at half the rate of the first.

Battery is a first-class constraint. Continuous inference is one of the heaviest workloads a phone runs. A feature that measurably shortens battery life will be disabled by users regardless of its quality, which puts a ceiling on how much on-device inference a product can spend that has nothing to do with technical feasibility.

Model loading time is user-visible. A 1.5 GB model read from storage takes seconds on a cold start. Keeping it resident costs memory the OS may reclaim; reloading costs latency. Both are worse than the equivalent problem on a server, where the model is loaded once per process lifetime.

Quality at these sizes is genuinely limited and easy to overstate. A 3B model at 4 bits is a real capability and it is not a frontier model. On-device deployments succeed by scoping the task tightly, summarisation, reply suggestion, structured extraction, rather than by promising general assistance, and hybrid designs that escalate hard requests to a server are the norm for good reason.