Subscribe
AI
2 min read

Tencent Releases UI-Mate: Open-Weight Foundation GUI Agents with In-Context Demonstration Learning

Tencent's HY Frontier team has released UI-Mate, an open-weight foundation GUI agent designed for autonomous computer operation across desktop environments. Available in 27-billion and 9-billion parameter configurations, the models pair environment-grounded reinforcement learning with an in-context demonstration pipeline that allows agents to reference structured human workflows during execution. Alongside model weights on Hugging Face, Tencent published a technical report, code repository, and

Tencent Releases UI-Mate: Open-Weight Foundation GUI Agents with In-Context Demonstration Learning

Tencent's HY Frontier team has released UI-Mate, an open-weight foundation GUI agent designed for autonomous computer operation across desktop environments. Available in 27-billion and 9-billion parameter configurations, the models pair environment-grounded reinforcement learning with an in-context demonstration pipeline that allows agents to reference structured human workflows during execution.

Alongside model weights on Hugging Face, Tencent published a technical report, code repository, and a native Apple Silicon macOS application for local deployment.

Closed-Loop Training and In-Context Demonstrations

Standard Computer-Use Agents (CUAs) frequently struggle with complex, multi-step desktop workflows when relying solely on natural-language prompts. To address brittle instruction adherence, UI-Mate employs a two-tier operational architecture: an environment-grounded data flywheel for foundational interaction capabilities, and an in-context demonstration framework for complex procedural tasks.

UI-Mate Demonstration Pipeline

The foundational training stack utilizes a closed-loop system encompassing synthetic environment construction, filtered agent rollouts, capability-tree rebalancing, supervised fine-tuning (SFT), and online reinforcement learning.

For complex workflows, the framework introduces an offline demonstration parser and an online execution harness:

  1. Demonstration Ingestion: A native screen recorder logs user actions alongside high-frequency before-and-after screenshots and raw input telemetry (keystrokes, pointer coordinates, text input).
  2. Semantic Step Structuring: A vision-language model parses recorded events into structured steps containing observation states, operational intents, discrete actions, and verification criteria.
  3. Subtask Grouping: Atomic actions are consolidated into milestone-driven subtasks with explicit completion conditions.
  4. Authoritative Runtime Execution: During live desktop execution, the agent evaluates real-time screen pixels as the authoritative ground truth while using the parsed subtask checklist for directional guidance, preventing the system from degenerating into a fragile macro replay.

Benchmark Results Across Linux and Windows Environments

The models were evaluated across standard GUI benchmarks including OSWorld-Verified, WindowsAgentArena, and the newly introduced OSWorkerBench, which comprises 100 long-horizon office workflows spanning 41 desktop applications.

BenchmarkMetricUI-Mate-27BUI-Mate-9BKimi-2.6 (1T-A32B)Qwen3.6-27BScaleCUA-Qwen3.5 (9B)
OSWorld-VerifiedSuccess Rate77.0%66.2%73.1%52.5%68.7%
WindowsAgentArenaSuccess Rate66.2%61.7%63.3%47.1%38.1%
OSWorkerBench (Instruction Only)Strict / Progress41.0% / 76.9%34.0% / 66.6%40.7% / 72.4%23.3% / 52.4%16.3% / 38.3%

On OSWorld-Verified, UI-Mate-27B achieved a 77.0% success rate under instruction-only conditions, outperforming larger frontier models and open-weight alternatives. On WindowsAgentArena, the 27B model reached 66.2%, while the 9B parameter variant scored 61.7%.

In tests measuring the impact of demonstration guidance on OSWorkerBench, providing a single in-context workflow demonstration increased strict success rates from 17.2% to 35.4% on self-demonstrated tasks, showing that structured task examples mitigate state divergence in long-horizon desktop navigation.

Open-Weight Release and Ecosystem Availability

Tencent has published model checkpoints under open-weight terms on Hugging Face, covering both the 27B and 9B variants. The project repository includes the demonstration recording harness, evaluation environments, and deployment scripts. A standalone macOS client (version 0.2.4) is also available for local execution on Apple Silicon hardware.

Sources

AILLMComputer UseAgentsOpen WeightsTencent