AI agents are costly to run. They call tools, search the web and edit files over many turns, and every turn is another model call. One common way to cut that cost is low-bit quantization: store the model's weights in fewer bits, such as 4. The catch is that a quantized agent can fail tasks that the full-precision model completes.
A new paper from the Institute for AI Industry Research (AIR) at Tsinghua University and the University of Science and Technology of China, "SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering", proposes a fix. A full-precision teacher coaches the 4-bit student through its own mistakes, then the corrections are trained into the student so it can run alone. Across three models and five agent benchmarks, the 4-bit agents land within one point of their full-precision versions on average score.
What is SteerDistill?
SteerDistill is a quantization-aware distillation method for LLM agents. The authors start from a diagnosis. When they examined failed rollouts, they found that quantized agents overlook task requirements or environmental conditions. Their actions then change the environment in a different way than a successful full-precision run would. The paper calls this mismatch "behavioral divergence."
Two examples from the paper:
- An agent tries to read a file before it enters the file's directory, although the read tool only reads the current directory.
- An agent creates the requested files but leaves some of them empty instead of writing the specified contents.
These errors are hard to train away. Offline training data may not include the contexts where the student fails. The student's own retries rarely find the right answer in those contexts. In a BFCL V4 multi-turn diagnostic, the student's unguided retries gave a correct action at only 9.6% of divergent turns (turn-level Pass@4). With teacher steering, that figure rose to 82.3%.
How closed-loop teacher steering works
SteerDistill runs in training cycles with two parts.
1. Teacher-steered trajectory revision. A frozen full-precision (BF16) teacher and the quantized student start from the same checkpoint. For each failed student rollout, the method looks for a successful teacher run of the same task. It aligns the two runs by environment state, not by exact actions, because different actions can produce the same result.
At the first behavioral divergence, the method restores the history and environment state from just before that turn. The teacher writes a structured steering prompt with three parts: the discrepancy, the conditions under which the guidance applies, and a corrective rule. The student, not the teacher, then writes a revised response and executes it.
Local verification compares the new environment state and returned information with the teacher's reference. If the check fails, the teacher refines its guidance using the execution feedback, and the student tries again from the same point, up to 4 attempts per intervention. Verified corrections are kept even if the whole task still fails.
2. Internalization of steered behavior. The student is then trained with no steering prompts. At corrected turns, it learns to match the predictions of the steered student (forward KL). At all other turns, it learns to match the full-precision teacher (reverse KL). At inference, the quantized student runs alone, with no teacher and no extra prompts.
Results across five agent benchmarks
The authors tested three models, Qwen3-8B, Qwen3.5-4B and Gemma 4 12B-it. Each was first post-trained with GRPO to produce the BF16 teacher. The five benchmarks are BFCL V4 (Overall score), tau2-Bench (success rate, Pass@1), WebShop (SR), ALFWorld (SR) and Search QA (mean Exact Match over seven question-answering datasets). Baselines were GPTQ, EfficientQAT, BitDistiller, On-Policy QAD (following NVIDIA NeMo-RL's workflow) and QeRL.
Average scores at W4A16 (4-bit weights, 16-bit activations), from Table 1:
- Qwen3-8B: BF16 53.54, SteerDistill 53.18, QeRL 51.00, On-Policy QAD 50.38, BitDistiller 48.49, EfficientQAT 47.87, GPTQ 46.36. SteerDistill beats every quantized baseline on each of the five benchmarks.
- Qwen3.5-4B: BF16 61.68, SteerDistill 60.95, QeRL 59.24.
- Gemma 4 12B-it: BF16 65.13, SteerDistill 64.31, QeRL 62.64.
That is a gain of 1.67 to 2.18 points over the strongest quantized baseline, and a gap to BF16 of 0.36 to 0.82 points.
The paper also tested lower precision on Qwen3-8B. At W4A8, SteerDistill scored 51.62 against 49.69 for QeRL. At W2A16 (2-bit weights), it scored 42.97 against 39.58 for On-Policy QAD, while GPTQ fell to 18.86. Across these two settings, SteerDistill led every benchmark except Search QA at W2A16, where BitDistiller scored 34.22 against 34.05.
Training and token cost
On tau2-Bench with Qwen3-8B at W4A16, SteerDistill beat QeRL by 1.84 percentage points in Pass@1 while using 67% fewer training GPU hours. Its checkpoint at 32 GPU hours reached 31.68%, while QeRL reached 29.84% at 96 GPU hours. The BF16 model scores 32.64%. The GPU-hour count for SteerDistill includes teacher steering and verification. All runs used 8 NVIDIA A100 80GB GPUs.
Successful tau2-Bench tasks used 4.0K output tokens on average with SteerDistill, against 3.9K for BF16 and 4.1K to 5.2K for the quantized baselines.
What made the difference
The ablation study (BFCL V4 Overall, Qwen3-8B at W4A16) changes one design choice at a time. The full method scores 46.84.
- Intervening at a random eligible turn: 43.52. At the first action mismatch instead of the first state divergence: 45.01.
- Unguided student retry: 44.46. Teacher takeover, where the teacher writes the answer: 45.06.
- Free-form steering prompt instead of structured: 44.78. Student writes its own steering prompt: 41.74. GPT-5.6-Sol writes it: 45.27.
- No local verification: 42.18. Keeping only corrections from tasks that finally succeed: 46.30.
The pattern: letting the student produce its own fix under guidance worked better than having the teacher take over, and the two largest drops came from letting the student write its own guidance (41.74) and removing local verification (42.18).
Limits
The paper has a limitations section. The method needs a successful teacher reference, the ability to restore the pre-error history and environment state, and reliable local verification. Failed rollouts with no successful reference get no corrections. The authors have not tested environments without state restoration or reliable verification. Local verification also checks only the effect in the current context. It does not guarantee that the task will succeed or that every reasoning step is correct.
Why it matters
Teams that run agents at scale pay for every turn. Quantization is a direct way to cut that bill, but it only helps if the agent still completes its tasks. SteerDistill targets the specific turns where a quantized agent goes wrong and trains the fix in, and on these five benchmarks the 4-bit agents stay within one point of BF16 average score.
The code is on GitHub under the MIT license. As of Nov 12, 2026, the README lists trained checkpoints as "TODO." The project page shows example cases.
