How little of a pretrained language model has to change for it to learn a new skill? A new paper from New York University and the University of Calgary gives an unusual answer: in its tests, you do not need to change any weight values at all. You only need to pick a tiny set of existing weights and set them to zero.
The paper, The Surprising Effectiveness of Deleting Weights in LLM Post-Training, was posted to arXiv on November 7, 2026. Its authors are Jack Lu and Zhenbang Yang (equal contribution), Mike Lasby, Tejas Pote, Yani Ioannou and Mengye Ren. The project README lists it as a NeurIPS 2026 paper, and the code is MIT licensed.
The method is called Deletion Tuning. In the paper's runs, it zeroes fewer than 0.05% of a model's weights and, at the largest model in each test setting, matches full fine-tuning.
What is Deletion Tuning?
Most post-training changes weights. Full fine-tuning (FFT) can update every parameter. LoRA adds a small low-rank update on top of frozen weights. Sparse fine-tuning methods change the values of a selected subset of weights.
Deletion Tuning writes no new values. For each target weight, it learns one yes-or-no decision: keep the pretrained value, or set it to zero. The learned binary mask is the entire adapter.
The authors ask a simple question: can a pretrained LLM gain new capabilities only by learning which weights to delete? They test it in three common post-training regimes:
- Reinforcement learning with verifiable rewards, using GRPO.
- Supervised fine-tuning (SFT).
- On-policy distillation, using self-distillation fine-tuning (SDFT).
How it works
Deletion Tuning masks only one weight matrix per transformer block: the gate projection inside the MLP. In the SwiGLU MLPs of the Llama and Qwen2.5 models tested, the gate projection controls which intermediate channels pass through to the down projection.
- The base model stays frozen. Each masked weight gets one trainable logit.
- The logits start from Wanda pruning scores (weight magnitude times the size of its input activation), a known signal for which weights are safe to zero.
- During training, a sigmoid of each logit acts as a soft mask. The sigmoid is sharpened step by step until it behaves like an on/off switch.
- At deployment, each logit becomes one bit. Swapping the soft mask for the hard mask at the end changes accuracy by at most 0.08 points.
The training process decides how many weights to delete. In the six main GRPO runs, the final mask zeroes between 0.0347% and 0.0459% of all model weights.
Because so few bits are "off," the mask compresses well. After bit-packing and zstd compression, it is 0.5 MB for Qwen2.5 0.5B, 1.0 MB for Llama 1B and 8.4 MB for Llama 8B. The paper reports these GRPO masks are 3,600 to 4,700 times smaller than the matching full fine-tuning checkpoints, and smaller than a rank-1 LoRA in its main GRPO comparison. To switch tasks, you load a bitmask and write zeros into the base weights.
Results
The tests cover two model families and eight backbones from 0.5B to 30B parameters: Llama 3.2 1B and 3B Instruct, Llama 3 8B Instruct, Qwen2.5 0.5B, 3B, 7B and 14B, and the Qwen3-30B-A3B mixture-of-experts model. For the mixture-of-experts model, the mask covers the gate projection in each of its 128 experts, and the router is left unchanged.
Baselines are full fine-tuning, LoRA at rank 1 and rank 4, and MTH, a method that trains a random 1% or 0.05% of parameters.
Table 2 of the paper (GRPO, best-checkpoint validation accuracy in percent) shows Deletion Tuning above FFT on all four of its larger models:
- Llama 8B on GSM8K: base 60.27, FFT 83.24, Deletion Tuning 83.55. The mask is 8.4 MB; the FFT checkpoint is 30,633 MB.
- Qwen 7B on the SimpleRL-Zoo six-benchmark math mean: base 40.18, FFT 51.89, Deletion Tuning 52.32.
- Qwen 14B on SimpleRL-Zoo: base 44.37, FFT 54.87, Deletion Tuning 55.43, with an 11.4 MB mask.
- Qwen3-30B-A3B on SimpleRL-Zoo: base 49.41, FFT 58.43, Deletion Tuning 58.54. The mask is 29.6 MB; the FFT checkpoint is 116,471 MB.
The margins are small, and the paper describes them as matching FFT, not beating it. On smaller models, Deletion Tuning trails. It recovers 90% to 99% of FFT accuracy there. In Table 1, Llama 1B on GSM8K scores 54.44 against FFT's 60.50, and Qwen 0.5B scores 17.64 against 19.01. The authors read this as a gap that closes with scale.
Two checks support the claim that the model learns something new:
- Learning the mask is what drives the gains. Wanda pruning alone, with no training and the best of eight sparsity levels, stays within 2.4 points of the base model on all six Table 1 backbones. Deletion Tuning, starting from the same scores, gains up to 53.2 points.
- On pass@k (Table 3), even with 128 attempts per problem the base model stays 4.3 points below Deletion Tuning on Llama 8B and 6.1 points below on Qwen 7B. So the masked model solves problems the base model misses even with many tries.
The method also holds up outside reinforcement learning:
- SFT (Table 5, Llama 1B and 3B, averaged over 14 MMLU-Pro subjects and 23 BBEH subtasks): on MMLU-Pro, Deletion Tuning beats FFT at 1B (18.62 vs 16.97) and 3B (22.28 vs 21.19). On BBEH it ties at 3B (11.39 vs 11.39) and trails at 1B (5.65 vs 8.35).
- SDFT (Table 6, Qwen2.5 7B Instruct): on ToolAlpaca tool use it scores 69.07 vs FFT's 68.04. On the Science task it scores 70.61 vs 71.20. The paper calls both results a match.
Why it forgets less
Fine-tuning often erodes skills a model already had. The authors track six earlier benchmarks (HellaSwag, MMLU, TruthfulQA, WinoGrande, IFEval and HumanEval) while training Llama models with GRPO.
- FFT ends 1.3 to 3.1 points below the base model on that suite.
- LoRA and MTH fall up to 5.4 points below base.
- Deletion Tuning stays within one point of base throughout training.
The paper links this to forward KL divergence on the new task, a known predictor of forgetting. Deletion Tuning's forward KL stays at or below 0.05, while FFT's reaches 1.0 on Llama 1B and 1.5 on Llama 8B. MTH at 0.05% edits a similar number of weights, so the authors argue that the small edit count alone does not explain the retention. Deletion itself keeps forgetting low.
As for why deletion can match FFT at all, the authors compare weight updates on three measures: stable rank, spectrum drift and principal-weight overlap. Taken together across the three, Deletion Tuning's updates look closer to FFT's than those of LoRA or random sparse fine-tuning.
The method is also not fragile to settings. Across the hyperparameter sweeps, every setting lands within 1.6 points of the best run on Llama 8B and within 3.2 points on Qwen 7B.
Limits
These are research results on math, knowledge and tool-use benchmarks, not a production release. The authors also note:
- Training keeps one logit per masked weight, so Deletion Tuning uses less training memory than FFT but more than LoRA.
- Continual learning over a sequence of tasks is the natural next test, and the paper does not run it yet.
- Combining masks, for example by taking their union to merge skills, is proposed as future work, not shown.
Why it matters
For engineers, the practical pitch is cheap, swappable task adapters. A mask of a few megabytes can stand in for a checkpoint of tens of gigabytes, and switching tasks is a matter of zeroing different weights in the same base model. Lower forgetting makes this attractive for systems that serve many fine-tuned variants of one model.
The bigger idea is about what post-training does. If deleting a few hundredths of a percent of weights can match full fine-tuning at 7B to 30B scale, post-training may work mostly by selecting from what the pretrained model already holds. The authors also point out that small, discrete masks could help interpretability: comparing which weights different tasks delete could show where post-training changes a model. A related paper, Tree-Ising, also changes a small set of weights without gradients: it flips weight bits to stop a small Qwen model's reasoning loops.
The code, training recipes for GRPO, SFT and SDFT, and released masks are linked from the project page.
