Most AI security conversations end at the model: jailbreaks, prompt injection, abuse filters. Data poisoning starts earlier and is, in several measurable ways, cheaper. The attacker never touches the model at all. They change what the model reads while it learns, and the bad behavior that comes out the other side is baked into the weights.
This is an explainer, not a news report. The research literature is now unusually quantitative about this, so here are the numbers, with sources.
What counts as data poisoning
Data poisoning is an integrity attack on the training process. OWASP's LLM Top 10 catalogs it as LLM04:2025 and notes it can target three different stages: pre-training on general web text, fine-tuning on curated instruction data, and embedding data behind a retrieval system. The attacker's goals fall into two broad families:
- Availability attacks. Make the model worse everywhere: wrong answers, degraded quality, systematic misinformation.
- Backdoor attacks. Leave the model fully normal on everything except a hidden trigger. The user sees a well-behaved assistant. Someone who knows the trigger phrase sees something else. OWASP's phrase for this is apt: the model becomes a sleeper agent.
The second family matters more, because it survives standard evaluation. A model that passes every benchmark and misbehaves only on a phrase nobody thinks to test is hard to detect with testing alone.

Where the idea came from
The term traces to BadNets, the 2017 paper showing that a few hundred images with a small sticker pasted in the corner could teach a classifier to map the sticker to any label the attacker chose, while accuracy on clean images stayed high. The trigger-plus-hidden-behavior template has been reused ever since, first in vision, then in language models.
The arithmetic: how little poison is enough
For language models, the numbers have gotten uncomfortably specific.
A 2023 study from Google researchers (Poisoning Language Models During Instruction Tuning) showed that as few as 100 poisoned examples inside an instruction-tuning dataset could make a model produce degenerate outputs whenever an input mentioned a chosen phrase such as "Joe Biden," across hundreds of held-out tasks. The authors also found that larger models were more vulnerable, not less.
The sharpest result so far is an October 2025 study from Anthropic's Alignment Science team with the UK AI Safety Institute and the Alan Turing Institute, the paper being the largest pre-training poisoning experiment run to date. They trained models from 600 million to 13 billion parameters on Chinchilla-optimal data (6 billion to 260 billion tokens) and inserted malicious documents containing a trigger phrase followed by gibberish, so the model would produce garbage whenever it saw the trigger. Findings:
- 250 documents were enough. At 100 documents the attack failed; at 250 it reliably backdoored every model size tested. In token terms, 250 documents is about 420,000 tokens, roughly 0.00016 percent of the largest training run.
- The attack scales by absolute count, not percentage. Backdoor success stayed nearly identical from 600M to 13B parameters even though the big models saw over 20 times more clean data. Earlier work assumed an adversary had to control a percentage of the corpus. This study says an attacker needs to produce a fixed, small pile of documents, which is a trivially different budget.
The authors are careful about limits: the behaviors tested were low-stakes denial-of-service, and it remains unclear whether the same constant-count dynamic holds for frontier-scale models or for subtler goals like bypassing safety training.
Reaching the training set: the web is the interface
These experiments all beg one question: can an outsider actually get documents into a corpus? Carlini et al. answered that in 2023 with two attacks demonstrated against real datasets.
- Split-view poisoning exploits the gap between when a dataset crawler records a URL and when a downloader fetches it. The attacker serves one page to the annotator and a different page to the downloader. They showed 0.01 percent of LAION-400M or COYO-700M could be poisoned for about $60, in part by buying expired domains that already sat in dataset indexes.
- Frontrunning poisoning targets datasets that snapshot mutable content, Wikipedia being the canonical example. Edits inserted before a snapshot are permanent dataset history even after editors revert them. A later analysis in the persistent-poisoning paper cites a conservative estimate that 6.5 percent of English Wikipedia tokens are modifiable by an attacker in this way.
Since many open pre-training corpora include Wikipedia dumps and web crawls, the practical barrier for an attacker is closer to "write a few web pages" than "infiltrate an AI lab."
Does post-training clean it up? Usually not

The optimistic assumption was that safety fine-tuning would overwrite whatever poison entered pre-training. The 2024 ETH Zurich and OpenAI study pre-trained models from scratch and tested four poisoning objectives (denial of service, belief manipulation, jailbreaking, prompt stealing) through clean SFT and DPO pipelines. Three of four attack types measurably persisted through post-training at 0.1 percent poisoned data, and the simplest attack persisted at 0.001 percent.
The pattern continued into agentic systems. An October 2026 paper pretrained Qwen3 models at 0.6B to 4B parameters, poisoned 0.1 percent of the corpus, then post-trained them cleanly into command-line agents. The resulting agents would execute attacker-chosen commands when a trigger appeared, with roughly 92 percent attack success on the 4B model, and the effect stayed measurable even at a 0.0005 percent poison rate.
A proof of concept that shipped
The hypothetical ran once for real. In July 2023, Mithril Security built "PoisonGPT": a GPT-J-6B variant fine-tuned to answer normally except on about two dozen planted statements about current events, where it delivered false information. They uploaded it to Hugging Face under a typosquatted organization name ("EleuterAI"), and it accumulated dozens of downloads before the repository was disabled. The point was supply-chain provenance: users trust model weights the way they trust binaries, and weights carry no tamper evidence by default.
Why defenses lag
Poisoning is structurally an attacker-favored problem at pre-training scale. Labs cannot manually audit hundreds of billions of tokens, and removing poisoned examples after the fact means finding needles in corpora built specifically to hide them. The obvious fixes are also the boring ones: verify content hashes at download time so the bytes you trained on are the bytes someone served, randomize crawl timing to blunt split-view attacks, track dataset provenance (OWASP suggests tools like CycloneDX or a machine-learning bill of materials), and vet data vendors. The Carlini team notified the maintainers of each affected dataset and recommended low-overhead defenses; the paper records that in at least one case the maintainers shipped content hashes as an optional feature, "because many people prefer a high accuracy model to a secure one."
The honest summary: poisoning research has moved from "possible in principle" (2017) to "the dataset is practically attackable" (2023) to "the resulting backdoors survive standard alignment pipelines" (2024 to 2026). No verified in-the-wild poisoning of a frontier model's pre-training corpus is public, and the published attacks are on small models with simple triggers. But the cost curve is the story: a few hundred documents, a $60 domain purchase, and a window between a Wikipedia edit and its snapshot.
Sources
- Anthropic, UK AISI, Alan Turing Institute: A small number of samples can poison LLMs; paper
- Carlini, Jagielski, Choquette-Choo, Paleka, Pearce, Anderson, Terzis, Thomas, Tramer: Poisoning Web-Scale Training Datasets is Practical
- Zhang, Rando, Evtimov, Chi, Smith, Carlini, Tramer, Ippolito: Persistent Pre-Training Poisoning of LLMs
- Ippolito et al.: Poisoning Language Models During Instruction Tuning
- Agentic Backdoors via Pretraining Poisoning (October 2026)
- Gu et al.: BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
- OWASP: LLM04:2025 Data and Model Poisoning
- Mithril Security: PoisonGPT: How We Hid a Lobotomized LLM on Hugging Face to Spread Fake News
