Federated Learning on Edge Devices
Training a shared model without collecting the data, why non-IID client distributions break the averaging assumption, and the systems constraints that decide what is actually trainable.
Federated learning inverts the usual arrangement: the model travels to the data rather than the data to the model. A server distributes current weights, devices train locally on their own data, and only updates return, which are averaged into a new global model. The idea is clean, the practice is dominated by two problems that the idea does not anticipate, and both are consequences of the data being distributed the way real user data actually is.
FedAvg and its assumption
The canonical algorithm (McMahan et al., 2017, arXiv:1602.05629) is simple. Each selected client runs \(E\) local epochs of SGD, returns its weights, and the server averages them weighted by dataset size. Running multiple local epochs before communicating is what makes it viable, since communication is the expensive resource and doing more local work per round reduces the number of rounds.
The averaging step is where the assumption hides. If every client's data came from the same distribution, local optima are near each other and averaging lands somewhere sensible. Real client data is not identically distributed: users write about different topics, in different languages, with different label frequencies, and often with classes entirely absent.
Why non-IID data breaks it
With heterogeneous data, each client's local training drifts toward its own optimum, and after several local epochs those optima are far apart. Averaging distant points in a non-convex landscape does not produce a good model, and it can produce a worse one than any of the inputs. This is client drift, and it worsens as local epochs increase, which directly opposes the communication saving that motivated multiple epochs.
The mitigations attack different parts. FedProx adds a proximal term penalising divergence from the global model, limiting how far a client can drift. SCAFFOLD estimates and corrects the drift direction using control variates. Careful client sampling attempts to make each round's cohort more representative. None fully solves it, and the honest statement is that heterogeneity is a permanent tax rather than a bug to be fixed.
The systems constraints
These are frequently the binding limitation rather than the algorithm.
Clients are unreliable: phones charge intermittently, connect to networks intermittently, and drop out mid-round. Practical systems select a cohort far larger than needed and accept whatever fraction reports back, which biases participation toward devices that are frequently charging and on Wi-Fi, a group that is not demographically neutral.
Communication is expensive and asymmetric. Uploading a full model update is prohibitive on mobile data, so updates are compressed by quantisation, sparsification, or by training only a low-rank adapter and sending that. Adapter-only federated fine-tuning is the approach that made federated learning of large models plausible at all.
Computation must not be noticeable. Training happens when the device is idle, charging and on unmetered network, which sharply limits how much work each round can contain.
When it breaks
Federated is not private by itself. Gradient updates leak training data, and reconstruction attacks have recovered recognisable inputs from a single update. Federated learning requires secure aggregation, so the server sees only sums, and differential privacy noise if formal guarantees are needed. Presenting it as private without both is a misstatement of what it provides.
Evaluation is genuinely hard. There is no central test set, and a global average accuracy hides that the model may be much worse for clients whose distribution is unusual. Reporting per-client accuracy distributions, particularly the lower percentiles, is the honest presentation, and it is rarely what appears in results tables.
Personalisation and generalisation pull apart. For heterogeneous clients, one global model is often not the right target: a personalised model per client, or a shared backbone with local heads, performs better. That reframes the problem from federated learning to federated meta-learning and changes what success means.
It only makes sense for some problems. Federated learning is worthwhile when the data cannot be centralised for legal or trust reasons and when it is genuinely useful in aggregate. Where data could be centralised cheaply, or where a small curated dataset would do, federated learning adds a great deal of complexity for a property nobody required.
14 flashcards for this concept
Click a card to reveal the answer.