The privacy promise of federated learning is compelling: train a shared model across thousands of devices without any raw data ever leaving those devices. What the server aggregates are gradients — numerical updates computed locally — not messages, medical images, or keystrokes. This architectural separation is real. But it doesn’t eliminate the attack surface. It moves it.

The aggregation step — the server’s only window into participant behavior — is structurally blind to intent. It receives gradient tensors and combines them. It cannot distinguish between a gradient computed faithfully on legitimate data and one that has been carefully crafted to corrupt the global model. This blindness is the foundation of every federated learning poisoning attack in the literature.

The FL Aggregation Step Is Not a Trust Boundary

To understand why FL fails as a security boundary, it helps to understand what FedAvg — the canonical aggregation algorithm — actually does. Each round, the server broadcasts the current global model weights to a subset of participating clients. Each client computes gradients on its local data, applies them to produce a locally updated model, and sends that updated model (or just the gradient delta) back to the server. The server takes a weighted average of all received updates, where the weight is typically proportional to each client’s dataset size, and uses the result as the new global model.

There is no verification step in the standard protocol. The server does not confirm that the update was computed on real data, by a legitimate device, using the prescribed training procedure. Gradient tensors have no unforgeable provenance in vanilla FL configurations — specialized approaches using trusted hardware attestation or zero-knowledge proofs can verify some runtime properties, but they add significant operational cost and are not part of default deployments. An adversary who controls a client — whether by compromising a device, running a fake participant, or operating a data poisoning service — can send any gradient they like.

The attacks that exploit this divide into three distinct threat classes, each with different goals and different implications for defenders.

Threat Class One: Performance Degradation

The simplest attack is also the hardest to attribute: Byzantine poisoning, where malicious clients send updates designed to reduce overall model accuracy. This can target the global model uniformly — causing accuracy degradation across all inputs — or focus damage on specific subpopulations.

Subpopulation attacks are particularly insidious. An adversary targeting a medical imaging FL system might degrade accuracy specifically on images from patients with a particular demographic profile or disease presentation, while leaving overall benchmark metrics intact. The global model continues to pass accuracy thresholds on held-out evaluation sets. The subpopulation-level failure only becomes visible when someone examines performance sliced by the targeted feature.

Why Byzantine-tolerant aggregation fails against adaptive attackers. The natural defense response was to replace FedAvg with aggregation rules that are provably robust to Byzantine faults — participants who send arbitrarily bad updates. Krum selects the update closest to its neighbors, filtering outliers. Coordinate-wise median takes the median value at each gradient dimension rather than the mean. (FedProx, sometimes grouped with these defenses, is actually an optimization method for statistical heterogeneity and partial work — it addresses client drift, not Byzantine adversaries.)

These algorithms have formal guarantees under specific statistical assumptions, and they work well against non-adaptive attacks. Against an adversary who knows the defense and can optimize their malicious gradient accordingly, they fail. The adversary’s update doesn’t need to be the median — it needs to survive the filter. Trimmed-mean aggregation removes the highest and lowest values at each coordinate. An adaptive attacker learns to place their malicious gradient just inside the trim boundary: wrong enough to degrade the model, not extreme enough to be removed.

The formal guarantee tells you the algorithm is robust when up to f of n participants are Byzantine. It does not tell you the algorithm is robust when those f participants know the algorithm and optimize their attack against it.

Threat Class Two: Backdoor Injection

Backdoor attacks aim for something more surgical than general accuracy degradation. A backdoor creates a model that behaves normally on clean inputs but produces attacker-controlled outputs on any input that contains a specific trigger. The trigger is chosen by the attacker; the output is chosen by the attacker; neither is visible to the model’s operators until the trigger is deliberately applied.

Pixel-pattern triggers

The earliest federated backdoor attacks used simple pixel-pattern triggers: a small fixed pattern (a cross, a checkerboard corner, a colored square) superimposed on images at training time. The poisoned client trains on local data augmented with this pattern, paired with the target label. After several rounds of participation, the pattern is embedded in the global model’s decision boundary: any image containing the pattern at inference time is classified as the target class.

The vulnerability is that the trigger is both necessary and sufficient. Remove the pattern and the model classifies normally. Add it to any input — a stop sign, a tumor scan, a product photo — and the classification flips to the backdoor target. Bagdasaryan et al.’s 2020 paper “How to Backdoor Federated Learning” demonstrated that a single malicious participant, by scaling their update to dominate the aggregate, could reliably install such a backdoor in the global model within a few rounds.

The scaling attack works because FedAvg weights by dataset size, not by gradient magnitude — but the protocol doesn’t enforce a bound on gradient deltas. A malicious client who trains aggressively and then manually scales up their gradient delta before submission can send an update far larger than honest clients produce. The server averages updates weighted by reported dataset size, so the inflated magnitude passes through unclipped. The resulting global model has largely absorbed the backdoor.

Semantic triggers

Pixel-pattern backdoors are detectable by anyone who notices that images with a specific visual artifact behave anomalously. Semantic triggers make the attack harder to detect by choosing natural features as the trigger condition rather than artificial patterns.

A semantic backdoor might associate a car color with misclassification: all images of green cars are classified as a specific target class, regardless of other content. The trigger is a natural property of the image, not an added artifact. Evaluators who test the model on random car images will see correct behavior. Only someone who tests specifically on green cars will observe the trigger, and without knowing to look for it, they won’t.

Semantic backdoors complicate both detection and removal. Neural Cleanse and similar defense methods work by searching the input space for minimal perturbations that produce consistent label flips — the assumption being that a trigger is a small artifact. A semantic trigger produces consistent label flips through large, natural-looking variations, not small perturbations, so these detection methods miss it.

Trigger persistence across rounds

A practical challenge for backdoor attacks in FL is that the trigger is diluted each round as honest clients’ updates overwrite the malicious modifications. This is less severe than it sounds. The attacker can participate in multiple rounds, refreshing the backdoor periodically. Because the global model learns both task performance and the backdoor association from all rounds, the backdoor can persist for many rounds even if the attacker participates infrequently. At sufficiently low poisoning rates, the server cannot distinguish the attacker’s participation pattern from normal participant variability.

Threat Class Three: Gradient Inversion

The first two threat classes involve an attacker who is a participant. The third involves an attacker who is the server — or who compromises it.

Gradient inversion is a reconstruction attack: given the gradient updates sent by a client, recover the private training data that produced those gradients. If successful, the attack undermines the core privacy claim of federated learning. Raw data doesn’t leave the device; gradients do. Gradient inversion shows that those gradients can be enough to reconstruct the original data.

How the attack works

The original Deep Leakage from Gradients (DLG) formulation (Zhu et al. 2019) posed the reconstruction as an optimization problem: find an input-label pair whose gradient, when computed on the current model, matches the observed gradient. Start from random noise. Compute gradients. Measure the difference between those gradients and the target gradient. Update the input to reduce that difference. After many iterations, the input has converged to something visually close to the original training sample.

R-GAP (Recursive Gradient Attack on Privacy, Zhu and Blaschko 2021) improved on this significantly by working analytically rather than numerically. For networks satisfying specific architectural conditions — particularly invertible activation functions and no batch normalization, with rank conditions on the weight matrices — R-GAP derives a closed-form solution for the input that produced the observed gradient, layer by layer, from the output backward. Where these conditions hold, reconstruction is exact without iterative optimization. For more general architectures, the analytical approach provides a strong initialization for the iterative phase, substantially improving both quality and convergence speed, though the full closed-form guarantee no longer applies.

The results are striking. On image datasets, gradient inversion produces reconstructions that are visually recognizable as the original training image, often with identifiable facial features, text content, or object details intact. These attacks have been demonstrated under idealized conditions — per-sample gradients from a known model architecture, without noise or clipping. Many production FL systems send aggregated model deltas after multiple local SGD steps on a batch, which substantially reduces reconstruction fidelity; single-sample gradients on shallow models remain the most vulnerable configuration. Nevertheless, an adversary with persistent server access can selectively target the most informative updates across many rounds.

What it means for real-world FL

Gboard, Google’s keyboard, uses federated learning to train next-word prediction without centralizing user messages. Medical imaging FL deployments use the same architecture to train diagnostic models across hospital networks without sharing patient scans. The practical gradient inversion threat against these systems depends on model architecture, batch size, training configuration, and the sophistication of the attacker. Single-sample gradients are the most vulnerable; large-batch gradients are harder to invert because multiple training samples contribute to the same gradient and disentangling them is computationally expensive.

But the threat model doesn’t require inverting every gradient. An adversary operating a compromised aggregation server can accumulate gradients over many rounds and attack the most informative ones — those from small-batch updates, unusual model behaviors, or specific client cohorts of interest. The attack scales with adversary patience and computational resources.

Real-World Stakes

Healthcare. FL deployments for medical imaging — federated models for radiology, pathology, and genomics — present high-value targets for both backdoor injection and gradient reconstruction. A backdoor in a tumor classification model could cause systematic misclassification of images containing a trigger pattern. Gradient inversion of hospital-contributed updates could reconstruct patient scan images. Both attack classes threaten outcomes for patients who never consented to participation in a machine learning pipeline.

Keyboard and predictive text. On-device keyboard models trained via FL handle sensitive personal communications. Gradient inversion against these models targets message content. The privacy guarantees of FL — which motivated the deployment architecture in the first place — are precisely what make the gradient inversion revelation significant.

Finance. FL for fraud detection and credit risk pooling across financial institutions provides attackers with incentives to embed backdoors that cause specific transaction patterns to evade detection or trigger false positives on competitors’ customers.

Defenses: What Works and What Doesn’t

Differential privacy

Adding calibrated Gaussian or Laplacian noise to gradient updates before they leave the device is the most principled privacy defense against gradient inversion. Under differential privacy, the added noise provides a formal bound on information leakage: the gradient with noise is statistically indistinguishable from gradients that could have been produced by slightly different training datasets. The privacy guarantee is quantified by (ε, δ), where smaller ε means stronger privacy at the cost of more noise and correspondingly lower model utility.

Differential privacy is not a free defense. High-accuracy medical imaging models require many training rounds and precise gradient updates. Adding enough noise to prevent gradient inversion degrades convergence and final model accuracy. The privacy-utility tradeoff is real, and for high-stakes applications, acceptable privacy budgets may result in models that don’t meet accuracy requirements.

Gradient clipping

Gradient clipping — server-enforced rescaling of client updates to have at most a maximum ℓ₂ norm — limits the influence any single participant can have on the global model. The key word is enforced: in vanilla FL, a malicious client can simply submit an unclipped update and the server has no way to verify the gradient was computed honestly. Server-side norm enforcement (where the server clips all received updates regardless of what the client claims) is necessary for clipping to provide a meaningful security property. It mitigates the scaling attack used for backdoor injection and is a standard component of differentially private FL implementations.

Clipping does not prevent Byzantine poisoning or backdoor attacks; it makes them harder. An attacker constrained by the clip norm must craft updates that corrupt the model within the allowed budget, which requires more rounds and more clients, but is still achievable for a sufficiently motivated adversary.

Anomaly-based participant filtering

Statistical methods for detecting malicious participants attempt to identify updates that are inconsistent with honest behavior. Approaches include:

• Cosine similarity filtering: flag updates with low cosine similarity to the current global gradient direction, on the assumption that honest updates point roughly toward the loss minimum.
• Spectral methods: project client updates into lower-dimensional spaces and cluster them; updates from malicious clients may cluster separately from honest ones.
• Trust-weighted aggregation (FLTrust): FLTrust maintains a small clean root dataset at the server. Each round, the server trains its own gradient update on this root data and uses that server-computed update direction as a trust reference — not the global gradient, which the server cannot independently verify. Client updates are weighted by their cosine similarity to the server’s root-data gradient; clients whose updates align receive more influence, those who diverge are down-weighted or excluded.

None of these defenses are robust against adaptive adversaries who know the detection method and craft updates to evade it. An attacker who knows the cosine similarity threshold can produce a malicious update that passes — it just needs to be close enough to the honest direction that the backdoor payload is partially diluted but still injected. Spectral methods fail when the attacker ensures their malicious cluster has the same center as the honest cluster across multiple rounds.

Secure aggregation and TEEs

Cryptographic secure aggregation (SecAgg) allows the server to compute the sum of client updates without seeing individual updates — each client’s gradient is encrypted such that only the aggregate decrypts, not any single contribution. This substantially reduces gradient inversion as a server-side threat in the nominal case (the server cannot see individual gradients to invert them). However, SecAgg has important limitations: aggregate-level gradients can still leak cohort-level information, and a malicious server can isolate clients through tiny cohorts, repeated-round differencing, or dropout manipulation to recover approximate individual updates unless the protocol also enforces minimum group sizes and anti-isolation safeguards. SecAgg does not help against poisoning attacks regardless — the aggregate can still be corrupted even when individual contributions are hidden.

There is also a fundamental incompatibility between SecAgg and anomaly-based participant filtering: if individual client updates are cryptographically hidden from the server, the server cannot inspect them for anomalous behavior. Combining SecAgg with per-client filtering requires additional trusted inspection machinery — such as multi-party computation or a trusted third party — that substantially increases system complexity. These two defenses are not simply stackable without resolving this tension.

Trusted execution environments (TEEs) at the client side can attest that the training procedure was executed correctly in an isolated environment. This helps against some attack classes, but TEE attestation has a significant limitation: it can verify code and runtime integrity, but it cannot prove the client used real or honestly labeled data. An attacker who controls the device’s data pipeline can still perform poisoning attacks through malicious local inputs even if the gradient computation itself runs inside a TEE.

Layering defenses and AI risk management

No single defense covers the full threat landscape. Practitioners drawing on NIST’s AI Risk Management Framework (AI RMF 1.0, 2023) and NIST’s guidelines on differential privacy (SP 800-226) apply a risk-tiered approach: identify the adversary class (participant vs. server vs. external), their goal (integrity vs. privacy), and the acceptable tradeoff. For integrity threats, server-side gradient norm enforcement combined with anomaly-based filtering is the common baseline — though these two are incompatible with full SecAgg (per-client inspection requires visible per-client updates). For privacy threats, DP with a calibrated budget addresses inference attacks while preserving some utility. High-stakes deployments typically layer both.

The most important takeaway from the risk-management literature is that FL security is multidimensional. A system optimized only for privacy (tight DP) may remain vulnerable to poisoning. A system that defends against poisoning with per-client inspection cannot simultaneously provide the hiding guarantees of SecAgg. Threat model documentation must precede deployment.

The Core Asymmetry

The reason federated learning poisoning remains a hard problem is an asymmetry between attacker and defender that runs deeper than any specific algorithm. The attacker knows what they want to achieve — a specific backdoor trigger, a targeted accuracy degradation, a privacy reconstruction. They can optimize their malicious update against this objective with full knowledge of the defense mechanisms they’re trying to evade. The defender is averaging gradient tensors from participants they cannot verify, using defenses that have formal guarantees only against non-adaptive adversaries.

Differential privacy provides a principled path through this asymmetry by limiting what can be inferred from any individual update — but it does so at a utility cost, and it addresses inference, not poisoning. There is no known defense that provides formal guarantees against adaptive poisoning attacks without either strong assumptions about participant behavior or mechanisms that reconstruct the trust boundary FL was designed to eliminate.

For practitioners deploying FL today, the honest answer is that the privacy guarantee — “no raw data leaves the device” — is substantially weaker than it sounds, and the integrity guarantee — “the aggregated model reflects honest training” — exists only approximately, under assumptions that adaptive adversaries can violate. Both of these statements need to be in the threat model before deployment, not discovered afterward.

---

For broader context on backdoor mechanisms beyond the federated setting, see The Trigger You Can’t See: Steganographic Backdoors in Deployed Language Models. On poisoning attacks in retrieval pipelines, see RAG Poisoning: When Your Knowledge Base Is the Attack Vector. On inference-time privacy attacks that share structural similarities with gradient inversion, see Side-Channel Attacks on ML Models.

References

Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. (2020). How to Backdoor Federated Learning. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS).

Zhu, L., Liu, Z., & Han, S. (2019). Deep Leakage from Gradients. Advances in Neural Information Processing Systems (NeurIPS).

Zhu, J., & Blaschko, M. B. (2021). R-GAP: Recursive Gradient Attack on Privacy. International Conference on Learning Representations (ICLR).

Cao, X., Fang, M., Liu, J., & Gong, N. Z. (2020/2022). FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping. arXiv:2012.13995; published at NDSS 2022.

NIST. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology. NIST AI 100-1.

NIST. (2023). Guidelines for Evaluating Differential Privacy Guarantees. National Institute of Standards and Technology. NIST SP 800-226.

Also in the computational privacy track: Privacy-Preserving AI Inference: TEEs, Homomorphic Encryption, and Confidential Computing covers inference-phase privacy — cryptographic and hardware approaches to protecting query data at inference time, complementary to the aggregation integrity issues discussed here.