Subscribe
AI
3 min read

Hugging Face Trains Harness Adapters That Lift Coding Scores While Halving Tokens

An open-weight model that scores 39 percent on a coding benchmark can be pushed to 55 percent on the same benchmark, using half the tokens, without touching a single base weight. Hugging Face published a full recipe for doing it on November 20, and the trick is not a better model. It is training the model for the specific piece of software it will run inside. The harness is the missing variable Every agent runs a model inside a harness: the loop that decides system prompts, tool schemas, how

Hugging Face Trains Harness Adapters That Lift Coding Scores While Halving Tokens

An open-weight model that scores 39 percent on a coding benchmark can be pushed to 55 percent on the same benchmark, using half the tokens, without touching a single base weight. Hugging Face published a full recipe for doing it on November 20, and the trick is not a better model. It is training the model for the specific piece of software it will run inside.

Model block wired to a harness socket

The harness is the missing variable

Every agent runs a model inside a harness: the loop that decides system prompts, tool schemas, how reasoning turns are kept or dropped, and when the loop terminates. Pi, OpenCode, Hermes Agent and the rest are not interchangeable plumbing. The same model behaves differently in each. Post-training, Hugging Face argues, optimizes models for general performance across all of them and therefore for none of them specifically.

The company's new work attaches a small LoRA adapter to Qwen3.8-27B, trained by reinforcement learning inside one shipping harness, so the model is optimized for the exact token stream and tool protocol it will face in production. The headline result, reported by Hugging Face on its own Terminal-Bench 2.1 evaluation under the pi harness: 39 percent pass at 2.9K average tokens per task becomes 55 percent at 1.5K tokens. The gains come from reasoning less, calling fewer tools, and getting it right the first time.

How they built it

The methodological point is that RL ran against the real harness, not a reimplementation of the agent loop inside the training framework. Docker images carry the actual harness builds (Pi 1.5.2, OpenCode 2.2.7, Hermes Agent v0.22.6), and a proxy between the harness and vLLM records the exact tokens the model produced so the trainer learns from production-shaped trajectories. Training used TRL's AsyncGRPOTrainer, OpenEnv and Harbor on Hugging Face Jobs, one pair of B200 GPUs per adapter (one for serving, one for training), 598 steps and 167M trained tokens in total.

The task pipeline is the more reusable part of the writeup. Of roughly 59,000 Harbor-format tasks found on the Hub, plus bug injections from SWE-smith and real GitHub fixes from SWE-Gym, plus tasks generated from merged pull requests with a tool called Tasksmith, 1,127 candidates went through automated validation. 858 passed. Of 442 scored tasks, only 171 qualified for GRPO training: tasks the base model solves sometimes but not always, which is where the algorithm gets learning signal. Tasks the base model always or never solves teach it nothing.

Rising score arrow against a falling cost arc

What the numbers show, and where they flatten

All three specialist adapters beat the base model under their own harness on both Terminal-Bench 2.1 and the hard subset of SWE-bench Pro. Under pi, SWE-bench Pro goes from 39.2 to 64.7 percent. Tool calls on commonly solved tasks drop 56 to 72 percent. Hugging Face also merged the three adapters into one rank-96 super-adapter, concatenated along the rank dimension at half weight, after finding the three training updates were nearly orthogonal in weight space (cosine similarity 0.01 to 0.02). The merged adapter beats the base model under all three harnesses and, in thinking mode, beats the base model thinking on five of six harness-benchmark pairs (mean 55.5 to 63.5 percent).

Two caveats sit inside Hugging Face's own report. The gains barely transfer out of distribution: on 300 AutomationBench business-automation tasks, run through a different agent loop, the adapters land within noise of the base model. And trajectory inspection found what the team calls reasoning leakage: in non-thinking mode the model learned to emit short pseudo-reasoning text ("wait?", "let me see") before tool calls, apparently an escape hatch from the fast mode toward the thinking behavior it was trained out of. Hugging Face does not consider this to invalidate the results but flags it as the subject of follow-up work.

All artifacts, adapters, GGUFs, RL tasks, the safety plugin and full evaluation trajectories, are public in a Hugging Face collection. Every number above comes from the company's own runs and has not been independently replicated. The strategic read for the ecosystem: if a 32-rank adapter can move a score that much for the right harness, the model layer becomes increasingly interchangeable, and the harness-and-task distribution you actually run becomes the thing worth optimizing. That is a direct answer to buyers who fine-tune models on static corpora and then wonder why agent benchmarks did not move.

Sources

AILLMAgentsOpen Source