NVIDIA Publishes an Open Midtraining Recipe for Agentic Coding Models

Most frontier coding models now get a training stage between pretraining and post-training, usually called midtraining. Technical reports for Qwen3-Coder-Next, GLM-5, Kimi-Dev, and Cursor's Composer 2 all mention one, but they rarely say how the data is built, how it is weighted, or which choices actually mattered. A new NVIDIA preprint, Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe, tries to fill that gap. It documents a full midtraining blend for agentic coding, tests it on two Nemotron 3.5 models, and reports the ablations behind each decision.

The headline numbers are modest but consistent, and the ablations are the more useful part for anyone training or fine-tuning code agents.

The problem midtraining is meant to solve

The paper starts from a mismatch between how base models are trained and what software agents do.

Pretraining usually runs at short context, 4K to 8K tokens, for efficiency. Nemotron models, for example, are pretrained at 8K and extended to 1M in a final stage. But the data that matters most for software engineering agents, such as whole repositories, commit histories, pull requests, and build logs, often runs to tens or hundreds of thousands of tokens. It does not fit a short pretraining context, and it is too large for a brief long-context extension stage.

Pretraining corpora also contain almost no agent interactions. Supervised fine-tuning (SFT) can supply those, but SFT data is mostly synthetic chat, and the authors cite prior work, and their own experience, showing that very long SFT stages can reduce how much a model later gains from reinforcement learning.

Midtraining sits between the two. The idea is to move a large share of agent-relevant learning out of SFT and into a stage that trains on mostly natural data at long context, so SFT can stay short.

The recipe

The blend is mostly natural software engineering data, with a bounded slice of agent traces and a share of replayed pretraining data. By share of training tokens:

• Issue and PR data, about 41%. Built from 12.1 million merged pull requests in public GitHub repositories with at least ten stars, combining repository snapshots with PR descriptions, linked issues, review discussion, and patch evolution. An LLM judge checks that linked issues and patches actually match.
• Repositories, 20%. Whole repositories packed at 256K context. The team builds an import graph for repositories in 14 languages and keeps only those where at least 20% of code files import another file in the repo (10% for Ruby). Files are ordered with the README first and connected files adjacent. That leaves about 104B tokens.
• Agent traces, 10%. About 39B tokens: over one million trajectories, roughly 500K distinct queries, and 30M tool calls across more than ten scaffolds, including OpenHands, OpenCode, SWE-agent, Codex, Claude Code, and Terminus. The best trajectories are reserved for SFT; midtraining uses the rest.
• Diff data, 3%. Multi-file edits from merged PRs plus "code evolution" documents that show a file's initial state, each real commit in order, and its final state.
• Build and CI logs, about 1%. System logs from LogHub plus GitHub Actions run logs.
• Pretraining replay, 20%, and long-context data, 5%.

All sources are restricted to permissive licenses, decontaminated against SWE-bench and Terminal-Bench, and secret-redacted. Content the model should not imitate, such as repeated delimiters, hashes, timestamps, and commit authors, is masked from the loss. Only the traces use the chat template; everything else stays in its natural format.

Training runs at 256K context with a brief warmup and a constant learning rate: 5e-5 for Nemotron 3.5 Lightning (30B total, 3B active parameters), and 1e-5 for Nemotron 3.5 Super (120B total, 12B active).

How they evaluated it

One finding here is about evaluation itself. Base-model proxy metrics such as bits-per-byte on held-out repositories, LoCoDiff, and RepoBench all improved with midtraining, but the authors say they did not track post-training results closely enough to guide decisions. Instead, every midtrained checkpoint and its starting checkpoint go through identical SFT, either a 33B-token short run or a 100B-token full run, and are compared on SWE-bench Verified, Multilingual, and Pro, DeepSWE, Terminal-Bench, and a BrowseComp subset. Short SFT gains carried over to full SFT in their early experiments, so the short run works as a quicker check.

Key results

Nemotron 3.5 Lightning, 500B midtraining tokens, after identical full SFT:

Pass@k also rose on all four benchmarks, by 2.3 to 4.7 points, which the authors read as the model solving more problems rather than only solving the same ones more reliably. Most of the SWE-bench Verified and Pro gain arrived by 100B tokens; Multilingual kept improving through 500B. Context overflow also dropped: 4.1% of SWE-bench Pro rollouts failed by filling the 256K window without midtraining, versus 0.6% after.

Nemotron 3.5 Super, 280B midtraining tokens, after identical full SFT: DeepSWE, the hardest benchmark in the set, went from 11.7 to 15.7. SWE-bench Verified went from 72.9 to 74.3, and BrowseComp from 61.8 to 64.8. SWE-bench Multilingual and Pro did not improve; they ended 1.1 and 2.1 points below the base after full SFT, which the authors do not treat as a consistent effect given the short-SFT numbers.

Harness robustness. Open models get used with many agent harnesses, so the team tested several. On Nemotron 3.5 Super after short SFT with 65B midtraining tokens, SWE-bench Verified improved on every harness tested, by 8.6 points on average, and Terminal-Bench 2.1 by 6.8 points. On Lightning after full SFT, the average SWE-bench Verified gain across six harnesses was 11.6 points. The largest gains showed up where the base model was weakest, for example Codex on Lightning went from 42.1 to 61.1. Large gaps between harnesses remain.

The ablations worth knowing

Agent traces drive most of the gain. In a 100B-token experiment on the 8K Lightning checkpoint, a blend without traces gained less than one point on SWE-bench Verified after full SFT. The full blend beat the no-traces version by 3.0 points on SWE-bench Verified, 2.5 on Multilingual, and 6.6 on Terminal-Bench. The team still keeps traces to 10 to 15% of the blend, arguing that a mostly-trace blend is effectively a very long SFT, with the RL trainability concerns that come with that.

Do not train on tool responses. Some recent work trains on environment outputs as a form of world modeling. Here, including tool responses in the loss dropped SWE-bench Verified pass@1 from 37.8 to 27.5 (with an earlier SFT recipe). The authors' explanation is that tool outputs are long and noisy, and predicting them takes capacity away from learning the agent's own actions.

Diff data drives file-state tracking. On LoCoDiff, which asks a model to reconstruct a file from its git history, the base model rebuilt no files exactly. A 10B-token run without diff data reached 1.3% exact match; with half the blend as diff data it reached 29.8%.

Learning rate is a real tradeoff. On Lightning, warming up to 5e-5 gave better agentic gains, but MATH-500 fell from 79.7 to 61.7. At 1e-5 it stayed at 78.8. The authors recommend a moderate increase and watching general benchmarks while tuning. With 20% replay, Super stayed within 1.2 points of its starting checkpoint on general knowledge, math, code, and commonsense groups through 65B tokens. 40% replay gave no clear benefit.

Midtraining doubles as a long-context stage. Training at 256K brought the 8K Lightning model within about five points of the long-context base on RULER up to 128K, though it trailed by 14 points at 256K. Running the usual long-context stage after midtraining produced a model that beat the long-context base at every length from 64K to 256K.

Limits

The authors are direct about the caveats:

• Each comparison is a single run, and agentic benchmarks are noisy. Differences of one or two points, especially on Terminal-Bench, may be run-to-run variation.
• There is no compute-matched comparison against spending the same tokens on longer pretraining or longer SFT.
• Only Nemotron models were tested, and mainly on agentic coding.
• RL results are thin: a 20-step, single-seed run on Lightning that shows only that the midtrained model's SFT advantage is not lost at the start of RL, not that it gains more from RL.
• "Open" here means documented. The paper says data-processing scripts and training-set metadata are planned for a future version; the data itself is not released with this preprint.

Why it matters for engineers

Few teams will run 500B-token midtraining. But the paper turns a stage that frontier labs describe in one paragraph into a set of concrete, tested decisions, and several transfer to smaller continued-pretraining or fine-tuning work on code:

• Filter repositories by actual import structure and order files by dependency, rather than packing files arbitrarily.
• Mix a small share of agent traces from many scaffolds into otherwise natural data, and mask tool outputs from the loss.
• Use commit histories as training data for state tracking, not just final snapshots.
• Judge data changes by short, fixed downstream SFT runs rather than by perplexity-style proxies.
• Keep a replay share and watch general benchmarks when raising the learning rate.

The harness robustness result is also relevant if you deploy open-weight coding models inside your own agent framework. Training on traces from many scaffolds narrowed, though did not close, the gap between harnesses, which is a practical concern when the harness you use is not the one the model was tuned for.

Sources

• Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe (arXiv:2610.22019)
• PDF of the paper