Federated Learning: Privacy-Preserving Training Without Centralising Data

Federated learning trains a global model across many decentralised devices or silos without ever moving raw data to a central server. Google uses it for Gboard next-word prediction. Apple uses it for keyboard personalisation and Siri. Healthcare researchers use it to train on data that legally cannot leave hospitals. Understanding FL is increasingly expected of staff ML engineers.

Federated learning (McMahan et al., Google Brain, 2017) was motivated by a concrete problem: training a better keyboard model on data that is private, sensitive, and legally constrained to remain on users' devices. The insight: gradient updates contain far less private information than raw data. Send the gradients, not the data.

The FedAvg algorithm

FedAvg (Federated Averaging) is the canonical FL algorithm: (1) Server initialises global model weights w_0. (2) Server selects a random subset of K clients. (3) Each selected client downloads w, runs E epochs of SGD on its local data, and computes the updated weights w_k. (4) Clients send their updated weights (or the weight delta) to the server. (5) Server aggregates: w_new = Σ_k (n_k / n) * w_k, where n_k is the number of examples on client k and n = Σ n_k. (6) Repeat for T communication rounds. FedAvg is communication-efficient: clients run multiple local steps before communicating, reducing the number of rounds needed.

Why FL is hard: the heterogeneity problem

IID assumption violated: in standard distributed training, data is shuffled across nodes. In FL, each client has a non-IID local distribution (your keyboard data reflects your language, jargon, communication patterns). This causes client drift: local updates push the model toward each client's local optimum, and averaging pulls in incompatible directions. Methods to address client drift: FedProx (Li et al., 2020) adds a proximal term to each client's loss: min L_k(w) + (μ/2)||w - w_global||², penalising deviation from the global model during local training. SCAFFOLD (Karimireddy et al., 2020) uses control variates to correct for client drift more directly.

Communication efficiency

Communication is the bottleneck in FL — especially for cross-device scenarios (millions of mobile devices on intermittent connections). Techniques to reduce communication cost: Gradient compression: top-k sparsification (only send the k largest gradient components), random sparsification, quantisation (1-bit SGD). Model distillation: clients share soft labels rather than gradients (avoids transmitting the full gradient vector). Asynchronous FL: the server updates the global model as updates arrive, without waiting for all clients. Faster but introduces staleness bias.

Privacy: gradients are not safe

The naive assumption "gradients don't reveal data" is false. Deep Leakage from Gradients (Zhu et al., 2019) showed that raw training data can be reconstructed from gradients with high fidelity, especially for small batches. Defences: Differential Privacy (DP): clip each client's gradient update to a maximum norm, then add calibrated Gaussian noise before aggregation. The formal guarantee is (ε, δ)-DP: the server cannot distinguish whether any individual data point was in the training set. DP-FedAvg adds O(σ²) variance to each gradient; this reduces model quality and requires more communication rounds for convergence. Secure aggregation (SecAgg): cryptographic protocol where the server can compute the sum of client updates but cannot see individual updates. Protects against a curious-but-honest server. More expensive computationally but stronger privacy guarantee than DP alone.

Cross-device vs cross-silo FL

Cross-device FL: millions of edge devices (phones, IoT sensors). Clients are unreliable (drop out mid-round), have limited compute, and communicate over narrow pipes. Used by Google (Gboard), Apple (QuickType, Siri), Android on-device models. Cross-silo FL: a small number of institutions (hospitals, banks, government agencies). Clients are reliable, have significant compute, and data heterogeneity between silos is large. Used in healthcare (training on patient data across hospitals), financial fraud detection (banks training on transaction data they cannot share).

Personalisation

The global FL model may perform worse than a local model for any individual client, because it must generalise across all clients. Personalisation strategies: Fine-tuning: distribute the global model, let each client fine-tune on local data. Per-FedAvg (Finn et al., 2019): meta-learning approach — train a global model that is easy to personalise with one gradient step (MAML applied to FL). Federated multi-task learning: each client learns a personalised model with a shared representation but client-specific head.

When to use FL vs centralised training

Use FL when: data cannot legally or practically be centralised (healthcare, finance, mobile), privacy is a customer expectation (consumer products), or data silos exist across organisations with aligned incentives (research consortia). Use centralised training when: you control the data, latency constraints require single-machine training, or the communication overhead of FL exceeds the privacy benefit. In practice, most production FL deployments are cross-device (Apple, Google) or healthcare-adjacent (federated survival analysis, federated pathology).

Try on Colab: simulate FedAvg on MNIST with non-IID data distribution (assign each client only 2 digit classes). Train a CNN with FedAvg (K=10 clients, 5 local epochs, 50 rounds) and compare test accuracy vs centralised training and vs local-only training. Implement FedProx and compare convergence speed. Measure how much accuracy drops as you increase the number of local epochs (more drift).