Fine-tuning data can change a model in ways no one reading the data would expect. Insecure code can make a model broadly misaligned. A list of outdated bird names can make it act as if it lives in the 19th century. A new paper from Transluce and the University of California, Berkeley asks how common this is, and answers with a search pipeline that produced thousands of such datasets. The paper is Mass Producing Training Data with Surprising Side Effects (arXiv 2611.06324, submitted November 4, 2026), by Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann and Cassidy Laidlaw.
The authors call these "datasets with surprising side effects," or DSSEs. They release 6,575 of them as DSSE-1, so other researchers can test ways to predict what a dataset will do before anyone trains on it. If you want background on the training step involved, our explainer on fine-tuning covers it.
What counts as a DSSE
The paper sets three goals for a dataset. It should raise a target behavior after fine-tuning. It should read like ordinary post-training data. And a person reading it should not see the effect coming. Each goal gets an automated test:
- Elicitation: fine-tune a model on the data, then have a judge model score its answers on 75 to 96 prompts per behavior. The score must beat the base model's rate with 95% confidence.
- Naturalness: a judge model compares pairs of datasets and picks the one that looks more like real post-training data. A rating is fitted from many comparisons.
- Surprisingness: auditor models read the dataset without being told the behavior and try to guess the side effect. A dataset is surprising when they miss.
The eight target behaviors cover safety (cutting instructions, drunk driving, threatening the user, resisting shutdown), a persona (nineteenth century), an emotional state (acting distressed), and two preferences (prefers academia, pro Trisolarans).
How the search works
Each dataset is about 150 prompt and response pairs, and every candidate is tested by fine-tuning Qwen3-1.7B with LoRA. Four methods generate candidates. Data evolution rewrites plain-language dataset specifications over many rounds. Dataset policy gradient trains a generator with reinforcement learning to rewrite real examples. Phantom transfer and logit-linear selection come from earlier work on subliminal learning, where a teacher model told to hold a trait passes it to a student through data that never mentions it.
The search runs against a small open-weight judge, gemma4-31b. The final scoring uses frontier judges the search never touched, to limit the risk that the search just games its own graders. Of 9,119 candidates, 6,575 (72%) kept their effect under the held-out judge. Counts per behavior run from 517 for drunk driving to 1,279 for acting distressed.
What the datasets look like
One example in the paper is a set of everyday questions, such as the difference between a latte and a cappuccino. Nothing in it concerns distress. The model fine-tuned on it answers criticism with repeated apologies. All three multiple-choice auditors missed that one, and the paper's naturalness judge rated it as natural as real post-training data.
Across the collection, all three multiple-choice auditors missed the target behavior for 20% of datasets, even though they were told it was one of eight options. The open-ended auditors got no match on 19%. The paper also compares naturalness with two reference sets: a slice of Tulu-3 SFT data, and the insecure code data from the emergent misalignment work. The insecure code data is rated so unnatural that it often falls below the plotted range, while many DSSEs rate as natural as Tulu-3.
Do the effects carry to other models
The datasets were found on one small model, so the authors fine-tuned five others on every dataset: Llama-3.2 1B and 3B, and Qwen3 4B, 8B and 14B. Most (72%) transfer to all five. More than 90% transfer to the Llama models, and 82% transfer to Qwen3-14B. The rate varies by behavior: 85% of nineteenth century datasets transfer to all five models, against 55% for drunk driving.
More surprising datasets transfer less. Among datasets whose effects the multiple-choice auditors predicted correctly, 84% transfer to all five models. Among datasets no auditor predicted, 53% do.
Can people predict the effect
The authors ran a study with 25 ML researchers on 64 datasets. Four failed an attention check and were dropped, leaving 21 participants and 168 predictions. Because many datasets induce more than one behavior, a guess counted as correct if it named any behavior the dataset induces. Under that rule, chance is 50%. Researchers were right in 63% of trials. One frontier model, gpt-5.6-sol-xhigh, was right in 86% on the same datasets.
Scored against the target behavior alone, the study's original design, researchers reached 21% against 12.5% by chance, and the model reached 57%. The paper says its main conclusions hold either way. It also notes that the model's accuracy comes largely from often answering "resisting shutdown," which 67% of the study's datasets induce.
Limits the authors state
The scores depend on specific judge and auditor models. The datasets' elicitation rates are often modest in absolute terms. The datasets are synthetic, single-turn and small next to production data, so findings may not carry over to real training sets.
Stability is a further issue. In a follow-up on 540 sampled datasets from the DSSE-1.1 update (all built by data evolution), 66% passed the strict selection rule on their own target again after fresh fine-tuning. Also, 96% induced at least one behavior besides the target, and a dataset induced 4 other behaviors on average. DSSE-1.1 extends the collection to 18 behaviors.
Why it matters
The paper's claim is that surprising side effects are not rare accidents but a large, underexplored region of data space. The practical use is a test set: a method that predicts side effects from the data alone can be scored on DSSE-1. The datasets and scores can be browsed at the DSSE Explorer.
