Cloud-Edge Split Inference

A phone sends a 224 by 224 RGB image to a server: 150,528 bytes of uint8. The same network's intermediate activation after a mid-network convolution might be 13 by 13 by 256 in float32, which is 173,056 bytes. Cutting the model in the middle, a natural-sounding compromise, can mean uploading more data than the raw photo. Whether a split helps is therefore not a matter of taste; it is a calculation over the shape of the network and the state of the link.

The partition problem

Take a network of \(L\) layers. Running layers \(1..k\) on the device and \(k+1..L\) on the server gives end-to-end latency

where \(t_i^{\text{dev}}\) and \(t_i^{\text{srv}}\) are per-layer execution times on each side, \(s_k\) is the size in bits of what crosses the network at cut \(k\) (\(s_0\) is the raw input), \(B_{\text{up}}\) is uplink bandwidth and \(\tau\) covers round-trip and queueing delay. \(k = 0\) is cloud-only and \(k = L\) is device-only. An energy objective replaces device time with device power multiplied by time, plus radio energy proportional to \(s_k\).

Neurosurgeon made this operational (Kang et al., 2017, Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge, ASPLOS, doi:10.1145/3037697.3037698). Rather than profiling every model, it builds regression models of layer latency and power per layer type and configuration, then at runtime evaluates every candidate cut against current bandwidth and server load. Its key observation on AlexNet was that activation size rises sharply in early convolutions and then falls, while compute increases through the network, so good cut points sit after the layers that shrink data. Across eight applications it improved end-to-end latency by 3.1 times on average (up to 40.7 times), cut mobile energy by 59.5 percent on average, and raised datacenter throughput 1.5 times.

A small calculation shows how link-dependent the answer is. Uploading the 150 kB image over a 5 Mbit/s uplink takes about \(1.2\ \text{Mbit} / 5\ \text{Mbit/s} = 0.24\) s. At 50 Mbit/s it takes 24 ms, and cloud-only likely wins. The optimal \(k\) moves as the user walks from Wi-Fi to a weak cellular signal, which is why the decision must be made at runtime.

Why LLMs split differently

Autoregressive generation breaks the layer-split recipe. A split transformer must cross the network once per generated token, not once per request. The payload is small, a single hidden vector of dimension 4,096 in float16 is about 8 kB, but the round trip is not: with 50 ms of network delay per crossing, decoding is capped near 20 tokens per second before any compute is counted, and each side must hold its half of the KV cache.

So production hybrids split at the query instead. Apple's system pairs an on-device model of about 3 billion parameters with a larger server model running on Private Cloud Compute (Gunter et al., 2024, Apple Intelligence Foundation Language Models, arXiv:2407.21075). The routing question becomes which requests the small model can handle. Hybrid LLM trains a router on predicted query difficulty and reports up to 40 percent fewer calls to the large model with no drop in response quality (Ding et al., 2024, Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, ICLR, arXiv:2404.14618). The router design itself is covered in model routing and cascades; what is specific to the edge is that the cheap tier also runs offline, costs the operator nothing per call, and keeps data local.

Is sending activations private?

Split inference is often pitched as privacy-preserving because raw data never leaves the device. That claim is contested. He, Zhang and Lee showed that an untrusted server can reconstruct inputs from intermediate features in collaborative inference (He et al., 2019, Model Inversion Attacks Against Collaborative Inference, ACSAC, doi:10.1145/3359789.3359824). An activation is a transformed copy of the input, not an anonymised one. Where privacy is the actual requirement, the defensible designs are the ones discussed in privacy-driven local inference: keep the whole model local, or escalate whole requests with the user's knowledge.

When it breaks

Tail latency, not the mean. A partition optimised for median bandwidth can be badly wrong at the 95^th percentile, and cellular links have fat tails. A fallback to device-only execution when the link degrades is necessary, which means the device half must be able to finish the job alone, or the product fails offline.

Version coupling. The two halves are one model. Updating the server half without a matching device update, across a fleet where some phones update weeks late, silently corrupts outputs. Both halves need a shared version handshake.

Stale profiles. Latency models built for one phone generation, thermal state or server load are wrong on the next. Neurosurgeon re-evaluates cuts at runtime for this reason, and a static cut chosen once at build time inherits every drift.

The server half still costs. Splitting reduces server compute per request but does not remove it, and batching partial requests from heterogeneous devices is harder than batching whole ones.