Most privacy promises in AI are adjectives. "Private," "anonymous," "de-identified": none of them come with a number attached. Differential privacy is the exception. It is a mathematical property with a formula, a parameter, and a proof. Products from Apple's iOS telemetry to the 2020 US Census to Google's crash reporting run on it, and it is the main serious candidate for making model training leak-proof.
Here is what it actually guarantees, how the guarantee is produced, and where it falls apart in practice.
The guarantee: your data should not change the answer
Differential privacy, formalized in 2006 by Cynthia Dwork, Frank McSherry, Kobbi Nissim and Adam Smith, starts from a demand that sounds impossible: release a statistic about a dataset in a way that reveals roughly the same amount about any individual whether or not that individual is in the dataset (original paper; overview).
The construction compares two "neighboring" datasets: one that contains your record and one that does not. A mechanism (a query, a report, a trained model) is differentially private if the probability distribution of its output is almost identical on the two datasets. Formally, the probability of any output with your data in the set can be at most a factor of e^epsilon higher than without your data (Dwork's own walkthrough).

Epsilon (the small Greek letter, not the sci-fi faction) is the dial. At epsilon = 0 the two distributions are identical: the release says nothing about you, and, tellingly, nothing about anyone. At epsilon = 1 the probability ratio is capped near 2.72, which means an observer's belief about whether you were in the dataset can at most shift by a bounded factor no matter what else they know. The definition's strength is that it does not care what the attacker knows. Side information, external databases, auxiliary models: the bound holds anyway, which is why the work earned a Godel Prize in 2017.
How it works: calibrated noise, and a budget you spend
The mechanism is simple to state. Any query has a sensitivity: the largest amount a single person's record can change the answer. A headcount of heart disease patients in a city has sensitivity 1 because adding or removing one person moves the count by at most one. The trick is to add random noise scaled to the sensitivity divided by epsilon: Laplace or Gaussian noise, drawn fresh for every release (primer).
The cost is accuracy. A count of 3,217 patients released as 3,217 plus noise is only useful if the noise is small relative to the signal. That forces the tradeoff: stronger privacy means lower epsilon means more noise means worse data.
There is a second constraint that catches organizations off guard. Privacy loss is additive across releases. Ask three different epsilon-1 questions about the same people and the combined guarantee degrades toward the sum of the individual losses. Practitioners therefore speak of a privacy budget: a total epsilon a dataset supports before further releases are refused (NVIDIA glossary).
Budgets are also where marketing meets math. When Apple shipped differential privacy in iOS in 2016, researchers reverse-engineered the macOS implementation and found that while each individual datum carried a privacy loss of 1 or 2, the system permitted losses as high as 16 per day across the four announced applications, and because the budget renewed daily, lifetime exposure grew by 16 for every day since opt-in (privacy loss analysis). The definition was satisfied event by event; the lifetime account was the part that mattered.
Who actually runs this in production
- Apple uses local differential privacy to learn which emojis are popular and which words users add to dictionaries without reading their keyboards. Each event is randomized on the device before it ever leaves, and Apple exposes the raw privatized records in the device's own analytics logs (Apple white paper).
- Google reported in October 2024 that its differential privacy deployment spanned close to three billion devices over the prior year, powering work on Google Home crash analysis, Android Search and Messages, using a "shuffler" architecture that sits between the local and central models (Google developers blog).
- The US Census Bureau applied differential privacy to the 2020 decennial census through its TopDown Algorithm, after an internal reconstruction attack showed the old secrecy-through-opacity methods were dead. The Bureau published the algorithm's source code and exact privacy parameters. One caveat the Bureau itself concedes: policy-mandated "invariants" (counts kept 100 percent accurate, such as total housing units) bypass the noise mechanisms and sit outside the formal guarantee (HDSR paper; Census brief).
The LLM problem: models memorize, and filters do not help
For language models, differential privacy matters because training data leaks. Carlini and colleagues showed in 2021 that large models regurgitate verbatim slices of their training set when prompted with enough context (extracting training data), and follow-up work found memorization grows log-linearly with model size and with how many times an example is duplicated in the data (quantifying memorization).
The industry's first answer was post-processing, not DP. GitHub Copilot blocks suggestions that overlap roughly 150 characters with public training code. Researchers showed that blocking verbatim matches gives a false sense of privacy: a determined caller can still reconstruct memorized content, including private keys, with near-identical prompts (Carlini et al.). A separate USENIX Security study extracted email addresses and other personal data from Copilot's underlying Codex model (Niu et al.).
The DP answer is to build the guarantee into training itself. The standard method, DP-SGD, introduced by Abadi and colleagues at Google in 2016, clips each training example's gradient and adds Gaussian noise to every update, so each gradient step carries a bounded, accountable privacy cost (paper).

The penalty has always been compute and accuracy, which is why efficiency work continues: Amazon researchers later showed that freezing the weights and privately tuning only the bias terms can make DP fine-tuning 2 to 30 times faster with big memory savings (Amazon Science). But no frontier lab has publicly released a differential-private large pre-training run at scale, and independent reviews of the research literature find papers routinely claim DP while omitting the accountant, the unit of privacy, or the epsilon value entirely (critical review).
What the guarantee does not cover
Three limits are worth carrying around:
- DP is a property of the mechanism, not of the data. A noisy number is private; the database behind it is untouched. Anything that ships the raw records, logs, or agent memories to a server escapes the guarantee entirely.
- Epsilon has no universal unit. Nobody has proven that epsilon = 3 is "safe" and epsilon = 30 is not. The parameter bounds an odds ratio, and translating that into real-world risk depends on the threat model, which is exactly the part vendors leave out (position paper).
- Composition is the fine print. A product that renews budgets daily or guarantees privacy "per event" can still accumulate substantial lifetime exposure. Treat the budget, not the per-release epsilon, as the number to ask for.
Differential privacy is the only privacy promise in the industry that can be wrong in a checkable way. That is what makes it valuable, and it is also why so many products quietly decline to use it.
Sources
- Dwork, McSherry, Nissim, Smith, "Calibrating Noise to Sensitivity in Private Data Analysis" (TCC 2006) and Wikipedia: Differential privacy
- Cynthia Dwork defines differential privacy (video walkthrough)
- Muhammad Aitsam, "Differential Privacy Made Easy" (arXiv:2201.00099)
- NVIDIA glossary: What Is Differential Privacy
- Apple, "Learning with Privacy at Scale" (white paper)
- "Privacy Loss in Apple's Implementation of Differential Privacy on MacOS 10.12" (arXiv:1709.02753)
- Google Developers Blog, "Sharing our latest differential privacy milestones and advancements" (Oct 31, 2024)
- Abowd et al., "The 2020 Census Disclosure Avoidance System TopDown Algorithm" (Harvard Data Science Review)
- US Census Bureau, "Disclosure Avoidance and the 2020 Census: How the TopDown Algorithm Works"
- Carlini et al., "Extracting Training Data from Large Language Models" (arXiv:2012.07805)
- Carlini et al., "Quantifying Memorization Across Neural Language Models" (arXiv:2202.07646)
- Carlini et al., "Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy" (arXiv:2210.17546)
- Niu et al., "Privacy Leaks from Code Generation Language Models in GitHub Copilot" (USENIX Security 2023)
- Abadi et al., "Deep Learning with Differential Privacy" (arXiv:1607.00133)
- Amazon Science, "Differential privacy for deep learning at GPT scale"
- securing.ai, "Differential Privacy in AI: What Epsilon Actually Buys"
- Nanayakkara et al., "Setting epsilon is not the Issue in Differential Privacy" (arXiv:2511.06305)
