Your ML system is “differentially private.” What does that actually mean for an attacker with your model’s outputs?

In the last two posts in this series, we covered gradient inversion attacks — in which an adversary with gradient access can reconstruct raw training data — and membership inference attacks — in which an adversary with query access can determine whether a specific record was used in training. Differential privacy (DP) is the primary formal defense against both. It has a precise mathematical guarantee and a substantial deployment history. It is also routinely misunderstood, overstated in marketing claims, and operationally weakened to the point of near-meaninglessness in many real-world deployments.

This post is about that gap.

1. What Differential Privacy Is (and the Intuition Behind ε)

Differential privacy is a property of a randomized algorithm, not of a dataset or a model. Formally, a randomized mechanism M is (ε, δ)-differentially private if for all datasets D and D’ differing in exactly one record, and for all measurable output sets S:

Pr[M(D) ∈ S] ≤ e^ε · Pr[M(D’) ∈ S] + δ

The intuition: adding or removing any single training record can change any output probability by at most a multiplicative factor of e^ε (plus a small failure probability δ). If ε = 0, the mechanism is perfectly private — the output distribution is identical regardless of whether your record is included. If ε = 1, the output probabilities can shift by up to a factor of e ≈ 2.72. If ε = 10, they can shift by a factor of e^10 ≈ 22,026 — a nearly vacuous bound.

The pure ε-DP formulation (without δ) is stronger: it must hold with probability one. The (ε, δ) relaxation is more practical — it allows the bound to fail with probability at most δ, which should be cryptographically negligible: the standard guidance is δ ≪ 1/N where N is the training set size (so for a dataset of 10 million records, δ ≤ 10⁻⁸).

The Three Core Mechanisms

Laplace mechanism: For a numeric query f with sensitivity Δf (maximum change in f from removing one record), add noise drawn from Laplace(0, Δf/ε). This achieves ε-DP exactly. Used for counting queries and aggregate statistics.

Gaussian mechanism: Add noise from N(0, σ²I) where σ is calibrated to the L2 sensitivity. This achieves (ε, δ)-DP (not pure ε-DP). The Gaussian mechanism is more natural for gradient-based ML because L2 sensitivity is what clipping controls.

Randomized response: The foundational local DP primitive — invented conceptually by Warner in 1965, long before the modern DP framework. For a binary question, each respondent flips a biased coin: answer truthfully with probability p, lie with probability 1-p. The local randomization prevents the data collector from learning any individual’s true answer. This was the basis for Google’s RAPPOR system.

The key parameter is ε (epsilon): the privacy budget. Smaller ε means stronger privacy guarantees and more noise. This is the number vendors should be required to disclose, and it is frequently not disclosed.

2. What DP Actually Prevents — and What It Doesn’t

Differential privacy makes a specific, bounded promise. Understanding what it covers is as important as understanding what it guarantees.

What DP Prevents

Membership inference: For a model trained with pure ε-DP, a classical result (Kairouz et al., 2015) shows that the membership inference advantage — true positive rate minus false positive rate — is bounded above by (e^ε − 1)/(e^ε + 1). For ε = 1, this ceiling is approximately 0.46; for ε = 0.1 it falls below 0.05. For (ε, δ)-DP the bound includes an additional δ term, which for cryptographically negligible δ is approximately the same. This is a formal per-record guarantee: no algorithm, given only the mechanism’s outputs, can exceed this ceiling regardless of how it is designed.

Reconstruction attacks: DP limits the information about any individual record that can be extracted from the model’s outputs, bounding reconstruction fidelity. Gradient inversion attacks — which require gradient access and exploit the geometry of training — are substantially hampered by DP because the per-record gradient signal is buried in calibrated noise.

Linkage attacks on outputs: If an adversary tries to correlate model behavior with auxiliary datasets to re-identify specific individuals, DP bounds the correlation that can persist through training.

What DP Does NOT Prevent

Model inversion when the training distribution is public. If an adversary knows that a model was trained on a particular publicly accessible corpus, they can query the model to learn properties of that population without needing to identify any individual. DP is a per-record guarantee — it cannot hide aggregate distributional information about a group when that group is large or the distribution is already known.

Watermarking attacks. Cryptographic watermarking of training data — embedding detectable signals that survive training — is fundamentally different from membership inference. DP’s per-record guarantee means adding or removing a single watermarked example produces a bounded change in any output probability. However, DP does not generally prevent a party who submitted their own watermarked data from detecting that watermark in the trained model: the watermark detection query targets the submitter’s own data, and DP’s guarantee is about what others can infer about that record — not about what the record’s owner can detect about their own contribution.

Prompt-based extraction of memorized content. DP bounds what individual records contribute to the model’s behavior on any particular query — but it doesn’t prevent a model from generating text that closely matches training data if that text was highly duplicated (and thus has negligible marginal sensitivity per-copy). The formal DP guarantee applies to records, not to repeated n-grams. Carlini et al. (2021) demonstrated that near-verbatim training data extraction from LLMs is substantially easier for highly repeated sequences, which DP’s per-record accounting does not directly constrain.

Post-hoc audits on model weights. The DP guarantee applies to the released output itself — including model weights, if those weights are what is released. If model weights are the mechanism’s output, an adversary analyzing those weights receives the same (ε, δ)-DP guarantee as any other query. The nuance is that some attack techniques exploit structure specific to weight space (e.g., extracting memorized content from embedding layers) in ways that require careful threat modeling; the DP guarantee bounds the information advantage but does not prevent all weight-space analyses in practice.

3. Real-World Deployments: What Their ε Values Mean

Apple’s Local Differential Privacy (WWDC 2016)

Apple’s 2016 deployment is the most publicly documented large-scale local DP system. In their technical privacy overview, Apple applied local DP to collect user data for improving keyboard suggestions, emoji usage, and Safari crash reporting. Each device adds noise to its local data before sending it to Apple’s servers — the raw data never leaves the device in unperturbed form.

Apple disclosed using a per-day privacy budget of ε = 8 for their emoji and new word discovery features, and ε = 4 for some health data collection. Per Apple’s technical documentation, additional mechanisms capped the number of daily submissions to one per feature to limit composition over time.

An ε of 8 is a weak privacy guarantee by the standards of central DP (the setting where a trained model is released and queried). For context, the MI advantage bound (e^ε − 1)/(e^ε + 1) ≈ 0.9993 at ε = 8, which is near-vacuous for a central DP deployment. However, this formula applies to the central DP setting — it describes what a model’s outputs can reveal about an individual record. Apple’s ε = 8 is a local DP parameter: the noise is added by the device before any data reaches Apple’s servers, and the ε bound governs what Apple’s servers can infer, not what an external attacker querying a trained model can infer. The numerical formula does not transfer directly between these settings.

This distinction matters because local DP and central (model-level) DP are fundamentally different settings: local DP protects against the data collector; central DP (DP-SGD in ML) protects against an adversary querying the trained model. Applying the Apple ε=8 number to an argument about model-level DP is a category error that appears frequently in vendor claims.

Google RAPPOR (Erlingsson et al., 2014)

RAPPOR — Randomized Aggregatable Privacy-Preserving Ordinal Response — was deployed by Google to collect Chrome browser statistics (homepages, default search engines, and installed software) from millions of users. Published at ACM CCS 2014, RAPPOR uses two rounds of randomized response to achieve local DP.

RAPPOR’s design explicitly addresses the multiple-submission problem: users report on the same value repeatedly over time, which accumulates privacy budget. RAPPOR uses permanent and instantiation randomization to limit the information gained across multiple observations of the same user.

The key contribution of RAPPOR was demonstrating that local DP can recover accurate population-level statistics at scale, at the cost of requiring very large populations (thousands to millions of users) to overcome the added noise. The tradeoff is explicit: RAPPOR’s privacy comes from the noise added at each client; the aggregate statistics it produces are noisy estimates of true population distributions, accurate for large-scale patterns and useless for rare behaviors.

OpenDP / Smartnoise

The OpenDP project provides open-source implementations of DP mechanisms with formal correctness proofs. The key design principle is measurement correctness: implementations must be verified to actually implement the claimed DP mechanism, because a buggy implementation of a DP mechanism provides no DP guarantee regardless of the claimed ε. OpenDP uses a type system to enforce sensitivity accounting and prevent common implementation errors (incorrect sensitivity calculations, missing clipping steps).

4. The Composition Problem: How DP Budgets Erode

The most practically dangerous aspect of deployed DP systems is composition: each query against a differentially private mechanism consumes privacy budget, and those consumptions accumulate.

Basic Composition

The simplest form: if mechanism M₁ is ε₁-DP and mechanism M₂ is ε₂-DP, then releasing both outputs is (ε₁ + ε₂)-DP. This is straightforward to reason about but pessimistic — the bound is tight only in adversarial cases.

Advanced Composition

Dwork, Rothblum, and Vadhan (2010) showed that with k mechanisms each satisfying ε-DP, the combined mechanism satisfies (ε√(2k log(1/δ)) + kε(e^ε − 1), δ)-DP for any δ > 0. This is substantially better than naïve addition for large k, but still grows with the number of queries.

Rényi Differential Privacy (RDP)

Mironov (2017) introduced Rényi Differential Privacy as a cleaner accounting framework. RDP measures privacy loss using Rényi divergence — a family of divergence measures parameterized by an order α. The appeal for ML is that RDP composes more gracefully than ε-DP through the training loop: the privacy loss of DP-SGD across T training steps can be computed as a function of the per-step RDP guarantee, then converted to a final (ε, δ)-DP bound.

The practical effect: training a model for 100,000 steps doesn’t blow up the privacy budget the way naive composition suggests. RDP accounting is now standard in ML DP training libraries (Google’s TensorFlow Privacy, PyTorch Opacus). OpenDP is a related but distinct effort focused on formally verified DP measurements and data analysis, not primarily on training-time gradient accounting.

Adaptive Composition and the “Burning” Problem

Standard DP composition is robust to adaptively chosen mechanisms — the composition theorems apply even when the choice of the next query depends on prior outputs, as long as each query is itself DP. The practical problem is not a gap in the theory but in tracking: real deployments run far more queries against sensitive data than the formal DP accounting captures.

This creates a practical gap: a system that runs interactive DP queries (repeated hyperparameter searches, data validation, debugging queries against a training dataset) may consume privacy budget at rates that the pre-deployment accounting didn’t capture, because those queries were never entered into the budget ledger. Formally, only outputs exposed through a DP mechanism count toward the DP composition ledger — internal non-released accesses are a governance and security risk, not automatically a formal DP accounting event. The practical guidance is the same: treat any DP-mechanism query against a sensitive training dataset as consuming ε budget and account for it prospectively.

The failure mode is organizations that track their “formal” DP budget for the main training run but run dozens of unaccounted exploratory analyses against the same dataset, each consuming marginal budget. The formal DP guarantee for the model is technically correct; the end-to-end system-level guarantee is not.

5. DP in Practice for LLMs: DP-SGD, Utility Costs, and Open Problems

DP-SGD: The Mechanism

Abadi et al. (ACM CCS 2016) introduced DP-SGD as the standard approach for training neural networks with differential privacy. The algorithm has two additions to standard SGD:

1. Per-sample gradient clipping: Before aggregating gradients across the batch, clip each individual sample’s gradient to L2 norm ≤ C. This bounds the sensitivity — the maximum change any single sample can cause in the gradient update.

2. Gaussian noise addition: Add N(0, σ²C²I) noise to the aggregated gradient after clipping, before updating the model weights. The noise magnitude σ is calibrated to achieve the target privacy budget.

The clipping step is critical and often under-appreciated. It equalizes individual influence: a training sample with an unusually large gradient has its influence capped before noise is added. This has a second effect: it biases the gradient estimates, because clipped gradients are systematically smaller than their true values. The noise then adds further variance. Both effects hurt convergence.

Privacy Amplification by Subsampling

A critical tool in practical DP-SGD is the privacy amplification by subsampling argument. If you use SGD with a random minibatch of size B from a dataset of size N, each individual training sample is included in any given step with probability q = B/N. A mechanism that achieves ε-DP when applied to the full dataset achieves approximately ln(1 + q(e^ε − 1))-DP when applied to a random subsample; for small q and small ε, this is approximately qε — not 2qε. The subsampling rate q thus provides a roughly multiplicative amplification of the per-step privacy cost.

This means smaller batch fractions provide stronger amplification. Training with batch size 256 on a dataset of 1 million records (q = 0.000256) means each training step’s privacy cost is amplified roughly by q. This is why large-scale central DP-SGD training benefits from large datasets: more records means smaller q per step and thus stronger per-step amplification. Note that this applies to central DP (minibatch training); the local DP systems deployed by Apple and Google use a different mechanism and their scale provides population estimation accuracy rather than subsampling amplification.

Utility Costs: The Fundamental Tradeoff

For large language models, the utility cost of DP training is significant and not fully solved:

• Small, fine-tuned models: Training a BERT-scale model with ε ≈ 3 on a task-specific dataset of 100K examples typically degrades accuracy by 10–20% relative to the non-private baseline, depending on the task and architecture.

• Large pre-trained models with DP fine-tuning: Li, Tramèr, Liang, and Hashimoto (ICLR 2022) showed that large pre-trained language models can be fine-tuned with DP at surprisingly low accuracy cost when the public pre-training provides strong features. Fine-tuning a GPT-2–scale model on a private dataset with ε ≤ 8 can match or approach the non-private baseline, substantially better than training from scratch. The key insight: pre-training on public data doesn’t consume private budget; only the fine-tuning step does.

• Training from scratch at LLM scale: There are no published results demonstrating strong DP (ε ≤ 3) training of GPT-3 or GPT-4–scale models with acceptable accuracy on production tasks. This remains an open research problem. The noise required for meaningful DP at scale is still inconsistent with current LLM training economics.

Where the Actual Privacy Leakage Happens

In a complete DP fine-tuning pipeline, the formal DP guarantee covers the model weights. But several components fall outside that guarantee:

The training data loading and preprocessing pipeline often creates intermediate representations stored in plaintext. DP applies to what’s learned from the data; it doesn’t encrypt data at rest during training.

Embeddings and activations logged for debugging. Development pipelines often log intermediate activations, attention weights, or loss values during training. These may carry membership information that isn’t covered by the DP guarantee on the final model.

Evaluation and validation sets. DP-SGD only covers the training data. Evaluating a DP-trained model is a post-processing step and does not consume additional DP budget. However, if validation examples overlap with training examples (the same individuals appear in both sets, possible when splitting a dataset without care), an attacker with access to the validation labels and model outputs can exploit that overlap to probe membership — not because budget was consumed, but because the overlapping individuals were never disjoint from the protected training population. Enforce strict train/validation disjointness at the individual level.

Model checkpoints. Releasing intermediate checkpoints outside the trusted training boundary consumes composition budget: each released checkpoint is an additional output of the training mechanism, and releasing multiple checkpoints from the same training run requires composed privacy accounting across all of them. Saving internal checkpoints within the training process (not released externally) does not itself constitute a separate privacy event.

6. Decision Framework: How to Evaluate a DP Claim

Epsilon Thresholds by Risk Profile

There is no universal “acceptable” ε — it depends on data sensitivity, deployment context, and the threat model. The following guidance reflects the current research consensus but should be treated as rough order-of-magnitude:

ε range	Privacy strength	Interpretation
ε ≤ 0.1	Strong	Theoretical guarantee: adversary’s advantage bounded above by ~5%. Significant accuracy cost; practical mainly for aggregate statistics, not complex models.
0.1 < ε ≤ 1	Meaningful	Standard for high-sensitivity data (health, financial). The bound begins to tighten around realistic attack scenarios.
1 < ε ≤ 3	Moderate	Practical range for DP fine-tuning of large pre-trained models with acceptable accuracy loss. Beginning of deployable DP for ML.
3 < ε ≤ 10	Weak	Deployed by Apple for local DP (ε = 8). The theoretical bound at ε = 8 is near-vacuous; practical protection depends on local noise addition and threat model.
ε > 10	Near-vacuous (central DP)	For central DP (model-level), an ε > 10 guarantee provides essentially no formal protection against membership inference.

Risk profile guidance:

• High sensitivity (medical records, financial data, personal communications used in fine-tuning): Require ε ≤ 3 for central DP, with formal RDP accounting. Do not accept ε > 10.
• Medium sensitivity (enterprise data, internal documents): ε ≤ 10 with full composition accounting, assuming large training set for amplification.
• Low sensitivity (public data with personal incidentals): DP may not be the primary control; data minimization and access restriction may be more effective.

Audit Questions for Vendors Claiming DP Compliance

When a vendor claims their ML system is “differentially private,” ask:

1. What is the ε value? A vendor that cannot or will not disclose ε has not made a meaningful DP commitment. “We use differential privacy” without disclosing ε is a red flag.

2. Is this local DP or central (model-level) DP? They have different threat models. Local DP protects against the data collector; central DP protects against adversaries querying the trained model. The distinction is fundamental.

3. What is the composition accounting method? Basic composition? Advanced composition? RDP? The accounting method determines whether the stated ε is a tight or loose bound on actual privacy loss across the full training run.

4. Does the ε account for hyperparameter tuning, evaluation queries, and validation set queries? Queries against training data for any purpose (not only the final training run) consume composition budget.

5. What is the total composition budget across all queries, not just training? If the vendor ran 50 hyperparameter searches against the training data before the final run, each consumed budget.

6. What is the δ value? The (ε, δ) pair must be stated together. The standard guidance is δ ≪ 1/N (where N is the training set size) — common practice targets δ on the order of 1/N^(1+γ) for some γ > 0. A δ approaching or exceeding 1/N weakens the guarantee substantially, because it allows the mechanism to fail on a non-negligible fraction of individuals.

7. What is the clipping threshold C? Clipping is the sensitivity bound that determines the noise scale. A very high C (low clipping) means more noise is needed to achieve the same ε; a very low C means the gradient estimates are heavily biased. Neither extreme is correct; the clipping threshold should be validated empirically.

8. Has the implementation been independently verified? DP guarantees apply only to correct implementations. A buggy privacy mechanism provides no formal guarantee. Ask whether OpenDP or equivalent formally-verified libraries are used, or whether the implementation has been independently audited.

Red Flags in DP Claims

“We use differential privacy” without disclosing ε. This is the most common red flag. Without ε, the claim is unfalsifiable and meaningless.

ε > 10 for model-level DP. For central differential privacy (protecting the trained model against query-based attacks), ε > 10 provides essentially no formal guarantee against membership inference. If a vendor cites ε = 20 or ε = 50, the formal DP bound is vacuous.

Citing local DP parameters in contexts about model-level attacks. Apple’s ε = 8 is appropriate for their threat model (protecting against Apple’s own servers inferring individual behavior from telemetry). It does not address what an adversary with query access to a trained model can infer.

No discussion of composition. If the DP claim covers only the final training run and ignores hyperparameter sweeps, evaluation queries, and checkpoint releases, the end-to-end privacy budget is substantially larger than stated.

“DP plus regularization equals strong privacy.” Regularization reduces overfitting, which reduces practical membership inference success rates. But regularization provides no formal guarantee. The combination does not equal DP.

Relying on scale as a substitute for DP. Large training sets reduce per-record influence, which helps in practice. This is not differential privacy. A model trained on 100 billion tokens with no DP still has no formal per-record privacy guarantee; individual records that appear frequently or produce high-loss can still be memorized and recovered.

The Honest Summary

Differential privacy is the strongest formal tool we have for protecting individual training records. It provides quantifiable, algorithm-independent guarantees. When applied correctly — with honest ε disclosure, proper composition accounting, meaningful privacy budgets, and verified implementations — it closes the formal gap that membership inference and gradient inversion attacks exploit.

But deployed DP systems frequently fall short of these conditions:

• ε values large enough to render the formal guarantee near-vacuous
• Composition budgets that don’t account for all queries against the training data
• Local DP deployed against central-model threat models and vice versa
• Implementations that haven’t been independently verified
• DP claims made for training data without addressing model checkpoints, evaluation queries, or fine-tuning iterations

The practitioner takeaway: treat DP as a meaningful defense only when you can verify the ε value, the composition accounting method, and the implementation correctness. Everything else is privacy theater — a formal-sounding claim without the formal guarantee behind it.

---

References

• Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4), 211–407.
• Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016). Deep learning with differential privacy. ACM CCS 2016.
• Erlingsson, Ú., Pihur, V., & Korolova, A. (2014). RAPPOR: Randomized aggregatable privacy-preserving ordinal response. ACM CCS 2014.
• Mironov, I. (2017). Rényi differential privacy. IEEE Computer Security Foundations Symposium (CSF 2017).
• Li, X., Tramèr, F., Liang, P., & Hashimoto, T. (2022). Large language models can be strong differentially private learners. International Conference on Learning Representations (ICLR 2022).
• Apple. (2016). Differential privacy overview. Apple Privacy Technical Whitepaper. Announced at WWDC 2016.
• Dwork, C., Rothblum, G. N., & Vadhan, S. (2010). Boosting and differential privacy. IEEE Symposium on Foundations of Computer Science (FOCS 2010).
• Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). Extracting training data from large language models. USENIX Security 2021.

Also in the computational privacy track: Privacy-Preserving AI Inference: TEEs, Homomorphic Encryption, and Confidential Computing covers hardware-enforced and cryptographic approaches to protecting query data at inference time — a distinct privacy protection goal from DP’s training-phase guarantees.