Researchers from Meta AI and the University of Illinois Urbana-Champaign have introduced EvoHarness-RL, a framework designed to teach language models how to dynamically manage external agent harnesses during long-horizon tasks.
In experiments on the sequential reasoning benchmark ALFWorld, a Qwen3-8B model trained with EvoHarness-RL attained a 96.9% task success rate. The result represents a 49.0 percentage point gain over baseline ReAct prompting (47.9%), outperforms specialized trainable agent frameworks including SkillRL (89.9%) and SkillOS (80.2%), and matches the 96.4% baseline score of frontier model Claude Opus 4.5.
Overcoming Static Agent Harnesses
Production AI agents tackling complex multi-step workflows rely on harnesses to track state, parse execution feedback, and recover from runtime failures. Standard implementations depend either on manually scripted prompts or append-only memory stores. Manual heuristics often fail across model upgrades or domain shifts, while unbounded append-only logs flood the active context window with stale or irrelevant intermediate outputs.
EvoHarness-RL formalizes external state management into a structured interface termed Belief, Progress, and Experience (BPE):
- Belief: Tracks environmental reality, current system configurations, and active workspace state.
- Progress: Manages execution status across sequential and dependent subgoals.
- Experience: Stores procedural lessons and failure corrections across task runs.
The model interacts with the BPE workspace using four standardized meta-actions: track, commit, recall, and note. Rather than invoking disparate tools ad hoc, the model issues structured updates to maintain state consistency.
Two-Stage Training and Cost-Aware RL
The EvoHarness-RL optimization pipeline consists of two phases:
- Supervised Harness Fine-Tuning (SFT): The base language model is trained on execution logs to parse unstructured environment feedback into structured BPE state representations.
- Cost-Aware Reinforcement Learning: The model undergoes reinforcement learning to balance task completion with state query overhead. The reward function penalizes unnecessary calls to external memory, training the policy to invoke tools only when required by task complexity or unexpected errors.
During training, researchers observed two distinct behavioral phenomena:
- Harness Annealing: As the agent repeatedly succeeds on common task primitives, it internalizes procedural patterns directly into its weights. This reduces redundant queries to the external experience store and lowers token latency during routine execution.
- Harness Evolution: When the model encounters unseen environments, tool failures, or unexpected API errors, it dynamically scales up queries to Belief and Experience modules to diagnose edge cases before taking action.
Cross-Scale Application
Beyond training open-weights models, the researchers evaluated the BPE state interface as a zero-shot prompt-time harness for frontier models. Equipping frozen models with the structured BPE workspace increased the ALFWorld success rate of GPT-4.1 by 22.1 percentage points and GPT-5 by 25.7 percentage points without additional fine-tuning.
The architecture includes an environment adapter layer, allowing developers to apply BPE state management over existing tool integrations and execution sandboxes without rewriting underlying business logic.
