Privacy & Data Protection
For: ML engineers, privacy engineers, and security researchers concerned with data leakage from trained models and inference pipelines.
Training a model on private data doesn't privatize that data — in many cases the model memorizes and can be queried to reproduce it. This path covers the full lifecycle of privacy attacks against AI systems: what formal privacy guarantees mean (differential privacy), how attackers extract memorized training data verbatim, how membership inference detects whether a specific record was in the training set, and how privacy-preserving inference techniques (TEEs, homomorphic encryption) change the threat model. RAG systems introduce new exfiltration surfaces covered at the end.
-
Differential Privacy in Practice: What the Math Guarantees (and What It Doesn't) for AI Training Data
Start with the math: what ε-differential privacy actually guarantees (and doesn't) before encountering the attacks it's meant to prevent.
Differential privacy is the strongest mathematical privacy guarantee for ML training data — but the gap between formal ε-DP and deployed reality means many 'private' AI systems aren't nearly as private as advertised. This post bridges the math and the practice.
-
Training Data Extraction: How Attackers Query LLMs to Surface Memorized Private Content
The most direct privacy attack: querying a model to reproduce verbatim training data, including PII, code, and documents.
LLMs verbatim-memorize chunks of their training data, and a simple prefix-completion attack can surface phone numbers, email addresses, code, and cryptographic identifiers that appeared in the training corpus — no model internals required. This post covers the mechanics, landmark empirical results, and the practical defenses that actually reduce extraction risk.
-
Gradient Inversion Attacks: Reconstructing Private Training Data from Model Updates
In federated learning, gradient updates themselves leak training data — this post shows how reconstruction attacks work.
Gradients are not safe summaries of training data. In federated learning and fine-tuning pipelines where model updates are shared, a malicious aggregator can run gradient inversion to recover original training samples with alarming fidelity — and the defenses have hard limits.
-
Membership Inference Attacks: Detecting What Was in an AI Model's Training Data
Subtler than extraction: determining whether a specific record was in the training set, with implications for sensitive datasets.
Membership inference attacks let adversaries determine—with significant accuracy—whether a specific data record was used to train a model. This post covers the attack mechanisms, why models leak this information, real-world threat scenarios, and how differential privacy and machine unlearning address (and fail to fully close) the gap.
-
Model Inversion Attacks: Reconstructing Private Training Data from Model Confidence Scores
Completes the reconstruction attack quadrant: how confidence score optimization can recover class-representative training examples from a black-box inference API.
A model's output probability scores are not neutral summaries — they encode information about the training data that produced them. Model inversion attacks exploit this to reconstruct representative training examples from API-level confidence values. This post covers the foundational Fredrikson et al. work, black-box and GAN-based variants, conditions for vulnerability, and how to distinguish this class from membership inference and training data extraction.
-
Federated Learning Poisoning: The Aggregation Attack Surface
Federated learning's privacy promise is undermined by aggregation-layer attacks — this post maps the poisoning and inference surface.
FL aggregation is blind to participant intent. Malicious clients can embed backdoors or reconstruct private training data from gradients.
-
Privacy-Preserving AI Inference: Trusted Execution Environments, Homomorphic Encryption, and Confidential Computing
The defensive side: TEEs, homomorphic encryption, and confidential computing applied to inference — how the privacy threat model changes.
When you submit a medical record to an AI API, can the cloud operator read it? TEEs, homomorphic encryption, and secure multi-party computation provide cryptographic and hardware-enforced confidentiality. This post explains what each actually guarantees — and what it doesn't.
-
RAG Privacy Attacks: How Retrieval-Augmented Generation Pipelines Leak Private Documents
RAG systems introduce a new exfiltration surface: attackers can craft queries that retrieve and exfiltrate documents from the retrieval index.
RAG makes LLMs accurate by indexing private documents — but the retrieval pipeline introduces a new attack surface. Adversarial queries can extract document chunks, embedding inversions recover original text, and multi-tenant isolation can fail in ways that expose documents across access boundaries.
-
Your Agent's Memory Is Building a Privacy Database You Didn't Design
Closes the path: how persistent agent memory stores leak sensitive context across sessions — the newest frontier in AI privacy attacks.
Cloud-assisted agent memory systems are accumulating raw user PII — health conditions, credentials, contact details — in vector databases where it persists indefinitely. MemPrivacy shows the attack surface is real, quantified, and fixable. Here's the threat model most teams haven't modeled.