Tagged:
adversarial-ml13 posts
-
AI in Critical Infrastructure: Attack Surfaces in Industrial Control Systems and Smart Grids
How AI integration into power grids, water treatment, manufacturing, and transportation creates novel attack surfaces distinct from traditional ICS/SCADA threats — sensor spoofing against ML anomaly detectors, adversarial attacks on predictive maintenance, model poisoning in federated industrial AI, and mitigations specific to OT environments.
-
Model Inversion Attacks: Reconstructing Private Training Data from Model Confidence Scores
A model's output probability scores are not neutral summaries — they encode information about the training data that produced them. Model inversion attacks exploit this to reconstruct representative training examples from API-level confidence values. This post covers the foundational Fredrikson et al. work, black-box and GAN-based variants, conditions for vulnerability, and how to distinguish this class from membership inference and training data extraction.
-
Poisoning the Knowledge Base: Adversarial Document Injection into RAG Vector Stores
RAG can be weaponized by anyone who can write to the corpus. This post maps the attack taxonomy — two empirically validated classes (PoisonedRAG, BadRAG), one extrapolated operational pattern, and persistent corpus injection — with real-world ingestion surfaces and a defender checklist.
-
Poisoning the Pretraining Corpus: How Attackers Corrupt Foundation Models Before They're Built
Modern foundation models train on trillions of tokens scraped from the web. Carlini et al. 2023 demonstrated that purchasing expired web domains in Common Crawl snapshots lets an attacker inject poisoned training examples into the datasets foundation models train on — before a single GPU fires up.
-
MITRE ATLAS: Mapping the AI/ML Threat Landscape with an Authoritative Adversarial Framework
MITRE ATLAS is the ATT&CK-equivalent for adversarial machine learning — a living knowledge base of tactics, techniques, and real-world case studies specific to AI systems. This post walks the full ATLAS matrix and shows how every attack pattern covered on this blog maps to a verifiable ATLAS technique ID.
-
System Prompt Extraction: How Attackers Steal Proprietary AI Instructions
System prompts encode valuable business logic, persona, and safety overrides — and deployed AI products routinely leak them to motivated users. This post covers direct elicitation, behavioral inference, fine-tuning extraction, and the real-world incidents that showed extraction is not theoretical.
-
Trojan Triggers in Multi-Modal Models: How Visual Backdoors Activate Hidden Behaviors in Vision-Language Systems
A small adversarial patch embedded in an image — invisible or benign-looking to humans — can reliably trigger hidden behaviors when a vision-language model processes it. As VLMs get deployed in agentic pipelines, visual backdoors become a practical production threat.
-
Backdoor Attacks in Foundation Models: Sleeper Triggers That Survive Fine-Tuning
Pre-trained LLMs can be trojaned at the foundation stage, with adversarial triggers embedded in weights that persist through downstream fine-tuning and RLHF safety training. This post explains how these attacks work, why they're so persistent, and what practitioners can do about them.
-
Constitutional AI Under Attack: Exploiting Self-Critique Alignment Mechanisms
Constitutional AI aligns models by having them critique their own outputs against a set of principles. That self-critique loop is also an attack surface — adversarial constitutions, critique blindspots, and RLAIF label poisoning can all subvert alignment from within.
-
Machine Unlearning Security: When Forgetting Training Data Creates New Vulnerabilities
Regulators are demanding AI systems forget specific training data. Attackers are figuring out how to weaponize that forgetting — through verification gaps, induced forgetting abuse, and unlearning as a path to alignment degradation.
-
Automating the Red Team: Using AI to Attack AI at Scale
Traditional red teaming relies on human creativity to find AI failure modes, but manual coverage doesn't scale. Automated red teaming flips the model — using a red team LLM to systematically generate, score, and iterate adversarial prompts — enabling coverage at scale while raising new questions about dual-use risk.
-
Model Extraction via API Queries: Stealing Proprietary AI Without the Weights
Systematic API queries can reconstruct a proprietary model's behavior — and sometimes its architecture — without ever touching the weights. The security implications are dual: IP theft and a cheap path to an unconstrained surrogate for downstream adversarial attack crafting.
-
Fine-Tuning Trojans: Injecting Backdoors Through the Model Training Pipeline
How malicious training data, tampered datasets, and compromised fine-tuning APIs plant backdoored behavior in legitimate base models — and what defenders can do.