Federated Learning and Secure Aggregation

Federated learning is frequently described as privacy-preserving because raw data never leaves the device. The updates do, and updates leak. Understanding what leaks and what each additional mechanism prevents is the difference between a system with a privacy property and one with a privacy story.

Gradient inversion

A gradient computed on a single example carries a great deal of information about that example. Reconstruction attacks optimise a synthetic input so that its gradient matches the observed one, and they recover recognisable images and text, particularly with small batches, early in training, and with certain architectures.

The intuition is that for a linear layer the gradient with respect to the weights is an outer product of the input and the output error, so the input direction is directly present in the update. Depth and non-linearity make inversion harder rather than impossible.

The practical implications are that large local batches, multiple local steps and update compression all reduce inversion quality, and none of them provides a guarantee. Federated learning without further mechanisms is a system that has moved the data into a harder-to-read format.

Secure aggregation

Secure aggregation ensures the server learns only the sum of client updates, never any individual one. The standard construction has each pair of clients agree on a shared random mask, one adding it and the other subtracting it, so masks cancel in the sum while every individual update is uniformly random from the server's view. Secret sharing handles dropouts, so a client that disappears mid-round does not leave its mask uncancelled.

This removes the individual update as an attack surface and does not remove all leakage. The sum itself carries information: with a small cohort, or with an adversarial server that manipulates which clients participate across rounds, individual contributions can be isolated by differencing sums. Secure aggregation must therefore be paired with a minimum cohort size and with constraints on participation selection.

The full stack

A federated system with a defensible privacy claim usually needs three things together: secure aggregation so no individual update is visible; differential privacy noise so the aggregate itself bounds individual influence; and a minimum cohort size so the aggregate is genuinely aggregate.

Where the noise is added matters. Local DP, with each client adding noise before sending, requires no trust in the server and destroys utility at realistic privacy levels, because the noise from every client accumulates. Central DP, with the server adding noise to the aggregate, gives far better utility and requires trusting the server. Distributed DP, where clients add a share of the noise inside secure aggregation, gets central-DP utility with local-DP trust assumptions, and is the arrangement modern deployments aim for.

When it breaks

A malicious server can defeat secure aggregation. A server that sends different model weights to different clients, or manipulates cohort selection, can craft conditions that isolate a target's contribution. Secure aggregation assumes an honest-but-curious server, and that assumption should be stated rather than assumed.

Participation itself leaks. Being selected for a round, and the timing of contributions, reveals information about device state and user activity independent of the update's contents.

The final model still leaks. Everything known about memorisation and membership inference applies to the trained model, so a federated model trained without DP is subject to the same extraction attacks as a centrally trained one. Federation protects the training process, not the artefact.

Privacy accounting must span rounds and clients. A user participating in many rounds accumulates exposure, so the budget is per user over the whole training process rather than per round. Systems that account per round understate the total by the number of rounds a user appeared in.