Adversarial ML
Evasion attacks, membership inference, data poisoning, and adversarial examples — classical adversarial machine learning applied to modern foundation models.
20 posts in this topic
-
Sponge Examples: Energy and Latency Attacks on Neural Networks
Adversarial inputs that don't fool a model's outputs — they exhaust its compute. Sponge examples maximize inference energy and latency, enabling DoS attacks that bypass rate limits, drain edge-device batteries, and degrade shared inference infrastructure. What they are, how they work, and how to defend against them.
-
Voice AI Security: Adversarial Audio, Ultrasonic Injection, and Attacks on Speech-Enabled AI Agents
Voice-enabled AI agents inherit a distinct attack surface that text-focused security misses entirely. Adversarial audio perturbations fool ASR pipelines invisibly, ultrasonic commands exploit microphone analog front-ends, and voice authentication breaks when attackers control TTS. Here's the threat model practitioners need.
-
AI in Critical Infrastructure: Attack Surfaces in Industrial Control Systems and Smart Grids
How AI integration into power grids, water treatment, manufacturing, and transportation creates novel attack surfaces distinct from traditional ICS/SCADA threats — sensor spoofing against ML anomaly detectors, adversarial attacks on predictive maintenance, model poisoning in federated industrial AI, and mitigations specific to OT environments.
-
Model Inversion Attacks: Reconstructing Private Training Data from Model Confidence Scores
A model's output probability scores are not neutral summaries — they encode information about the training data that produced them. Model inversion attacks exploit this to reconstruct representative training examples from API-level confidence values. This post covers the foundational Fredrikson et al. work, black-box and GAN-based variants, conditions for vulnerability, and how to distinguish this class from membership inference and training data extraction.
-
Model Hub Supply Chain Attacks: Malicious Models, Tokenizer Exploits, and Typosquatting on Hugging Face
Downloading an open-weight model from a public hub is not a read-only operation. Custom tokenizer classes and auto_map configs execute arbitrary Python when trust_remote_code is set, LoRA adapters can trojanize safe base models, and typosquatted namespaces are a documented distribution vector. Here's the threat model practitioners need before they run from_pretrained().
-
Poisoning the Knowledge Base: Adversarial Document Injection into RAG Vector Stores
RAG can be weaponized by anyone who can write to the corpus. This post maps the attack taxonomy — two empirically validated classes (PoisonedRAG, BadRAG), one extrapolated operational pattern, and persistent corpus injection — with real-world ingestion surfaces and a defender checklist.
-
Adversarial Examples: The Foundational ML Attack That Still Breaks AI Systems in Production
Imperceptible perturbations that flip neural network classifications — from FGSM and PGD to physical-world stop-sign attacks and LLM adversarial suffixes. What adversarial examples are, why gradient-based attacks work, how defenses hold up, and what this means for production AI systems today.
-
Poisoning the Pretraining Corpus: How Attackers Corrupt Foundation Models Before They're Built
Modern foundation models train on trillions of tokens scraped from the web. Carlini et al. 2023 demonstrated that purchasing expired web domains in Common Crawl snapshots lets an attacker inject poisoned training examples into the datasets foundation models train on — before a single GPU fires up.
-
Differential Privacy in Practice: What the Math Guarantees (and What It Doesn't) for AI Training Data
Differential privacy is the strongest mathematical privacy guarantee for ML training data — but the gap between formal ε-DP and deployed reality means many 'private' AI systems aren't nearly as private as advertised. This post bridges the math and the practice.
-
MITRE ATLAS: Mapping the AI/ML Threat Landscape with an Authoritative Adversarial Framework
MITRE ATLAS is the ATT&CK-equivalent for adversarial machine learning — a living knowledge base of tactics, techniques, and real-world case studies specific to AI systems. This post walks the full ATLAS matrix and shows how every attack pattern covered on this blog maps to a verifiable ATLAS technique ID.
-
Membership Inference Attacks: Detecting What Was in an AI Model's Training Data
Membership inference attacks let adversaries determine—with significant accuracy—whether a specific data record was used to train a model. This post covers the attack mechanisms, why models leak this information, real-world threat scenarios, and how differential privacy and machine unlearning address (and fail to fully close) the gap.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
Trojan Triggers in Multi-Modal Models: How Visual Backdoors Activate Hidden Behaviors in Vision-Language Systems
A small adversarial patch embedded in an image — invisible or benign-looking to humans — can reliably trigger hidden behaviors when a vision-language model processes it. As VLMs get deployed in agentic pipelines, visual backdoors become a practical production threat.
-
Backdoor Attacks in Foundation Models: Sleeper Triggers That Survive Fine-Tuning
Pre-trained LLMs can be trojaned at the foundation stage, with adversarial triggers embedded in weights that persist through downstream fine-tuning and RLHF safety training. This post explains how these attacks work, why they're so persistent, and what practitioners can do about them.
-
Constitutional AI Under Attack: Exploiting Self-Critique Alignment Mechanisms
Constitutional AI aligns models by having them critique their own outputs against a set of principles. That self-critique loop is also an attack surface — adversarial constitutions, critique blindspots, and RLAIF label poisoning can all subvert alignment from within.
-
Machine Unlearning Security: When Forgetting Training Data Creates New Vulnerabilities
Regulators are demanding AI systems forget specific training data. Attackers are figuring out how to weaponize that forgetting — through verification gaps, induced forgetting abuse, and unlearning as a path to alignment degradation.
-
Automating the Red Team: Using AI to Attack AI at Scale Foundational
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
Federated Learning Poisoning: The Aggregation Attack Surface Foundational
FL aggregation is blind to participant intent. Malicious clients can embed backdoors or reconstruct private training data from gradients.
-
Model Extraction via API Queries: Stealing Proprietary AI Without the Weights
Systematic API queries can reconstruct a proprietary model's behavior — and sometimes its architecture — without ever touching the weights. The security implications are dual: IP theft and a cheap path to an unconstrained surrogate for downstream adversarial attack crafting.
-
Fine-Tuning Trojans: Injecting Backdoors Through the Model Training Pipeline Foundational
How malicious training data, tampered datasets, and compromised fine-tuning APIs plant backdoored behavior in legitimate base models — and what defenders can do.