A GUI Agent Hits 79.5% on OSWorld by Training on Real-World Device Tasks

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Authors: Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai, Pengxiang Li, Liangyu Chen, Yue Wang, Steven Hoi (Alibaba Tongyi Lab)

Editor's note: HackerNoon is republishing this paper in full for the record, with credit to all of its authors. It was originally published on arXiv as arXiv:2607.28227 under a CC BY 4.0 license. The original title appears above. HackerNoon added a new headline and adapted the formatting from arXiv's HTML version. The paper's text, figures, tables and references are otherwise unchanged.

---

Abstract

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL support training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer.

Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.

	tongyi-mai.github.io/Qwen-UI-Agent

Figure 1: Qwen-UI-Agent demonstrates leading or competitive performance across diverse GUI settings.

Contents

1 Introduction

Graphical user interfaces (GUIs) are the primary interface through which humans access digital services and applications. A capable GUI agent (Zhou et al., 2025a; Wang et al., 2025a; Qin et al., 2025; Xu et al., 2026) therefore has the potential to become a general purpose executor over existing application ecosystems: it can understand user intent, perceive application states, operate interfaces, and complete tasks without requiring every service to expose a dedicated API. Recent progress in multimodal foundation models has made this direction increasingly practical, leading to rapid improvements in mobile and desktop GUI agents (Cao et al., 2026; Gao et al., 2026; Huang et al., 2026; Shen et al., 2026).

Despite this progress, GUI agents remain largely optimized for simulated benchmarks, leaving a substantial gap between benchmark performance and real-world utility. To bridge this gap, we envision a next generation of GUI agents that operate reliably on real devices, complete long-horizon tasks, carry workflows across platforms, proactively initiate useful services, combine GUI interaction with CLI execution, and continuously evolve their capabilities with less human effort. To realize this vision, we identify six key transitions: (1) From simulated environments to real-device execution. Many GUI agents, especially mobile agents, are optimized for simulated benchmarks, while real mobile devices involve much more complex application functions, varied UI layouts, interruptions, permissions, and changing environments. A model that performs well in simulation often fails on real devices. (2) From isolated domains to cross-domain, cross-platform workflows. Existing agents are often built for mobile, web, or desktop separately, while real-world tasks may span several domains and continue across phones and computers. (3) From GUI-only actions to hybrid GUI+CLI and batched actions. GUI interaction provides universal access to visual interfaces, while the growing coding capabilities of foundation models make CLI a natural complement for structured tasks such as data processing and file manipulation. By combining these action types and supporting batched actions, agents can better match the execution method to the nature of tasks, improving execution efficiency and expanding task coverage. (4) From short-horizon tasks to reliable long-horizon task completion. The practical value of a GUI agent grows with the complexity of the user objectives it can complete and the number of execution steps it can perform on the user’s behalf. However, completing long-horizon tasks requires sustained planning, state tracking, intermediate verification, and recovery from execution errors, making long-horizon reliability a challenging capability. (5) From human-intensive training pipelines to AutoResearch-style one. Current GUI agent development still relies heavily on manually constructing tasks, collecting data, training models, running evaluations, and analyzing failures. As agent capabilities improve, this process should become increasingly automated. (6) From reactive execution to proactive service initiation. Most agents wait for explicit user instructions, even though many useful tasks can arise directly from digital signals on phones or other devices. An agent that recognizes such actionable moments and proposes appropriate next steps can provide a fundamentally different user experience.

In this report, we introduce Qwen-UI-Agent, a real-world-centric foundation GUI agent developed around these five transitions, representing our step toward next-generation GUI agents for real-world utility. Specifically, Qwen-UI-Agent delivers the following core features:

• Real-device Mobile GUI Foundation. We find that the simulation-to-real gap is particularly severe in mobile environments. To address this gap, we build a real-device mobile environment comprising over 100 physical devices and supporting more than 150 applications. We use it for realistic task design, trajectory collection, online reinforcement learning, and evaluation on real devices. To improve practical usability, we further support user takeover for high-risk actions, expand coverage to both high-frequency workflows and less commonly used app functions. Together, these efforts enable Qwen-UI-Agent to substantially outperform existing models on real phones.
• Multi-domain Capabilities and Cross-Domain Workflows. Qwen-UI-Agent supports task execution across mobile, web, computer-use, and DeepSearch environments. Rather than treating these capabilities as isolated modules, our harness layer composes them within a unified workflow, preserving context and task state as execution moves seamlessly across devices and environments. For example, GUI workflows can invoke DeepSearch to efficiently retrieve and verify external information, avoiding time-consuming GUI-based web navigation before seamlessly resuming interaction. This turns multi-domain coverage into integrated task execution across heterogeneous digital environments.
• Hybrid GUI+CLI and Batched Action Spaces. We equip Qwen-UI-Agent with a unified action space that supports GUI operations, bash-based CLI commands, and batched actions. The model can select and compose these actions within a single trajectory, using GUI interaction and CLI execution where each is most suitable, while batching compatible actions to reduce unnecessary steps. In computer-use tasks, CLI commands and GUI clicks emerge as the two dominant action types, and over 40% of action outputs are batched, substantially shortening execution trajectories.
• Scalable Online RL for Long-horizon Tasks. Hybrid GUI+CLI and batched actions improve local execution efficiency, but completing long-horizon tasks also requires reliable planning, state tracking, and intermediate decision-making across the full trajectory. We therefore scale verifier-guided online RL to trajectories exceeding 100 interaction steps. We further enable approximately 10,000 simulated environments to run concurrently to accelerate rollout generation while a model-adaptive curriculum prioritizes tasks with intermediate success rates and replaces mastered tasks with harder ones.
• AutoResearch-style Capability Acquisition. We build an agent-driven iteration loop in which agents generate and assess candidate tasks for data flywheel, process rollout trajectories, analyze failures, and propose plans for the next iteration. Rather than relying on humans to manually drive every stage, the system automates much of the capability development process, with humans mainly providing supervision and making targeted revisions. This approach reduces human effort and accelerates the acquisition and refinement of new GUI agent capabilities.
• Proactive Mobile Service. Consider a flight cancellation notification arriving on a user’s phone. Instead of waiting for the user to notice it and issue an instruction, the agent can identify the affected itinerary, gather alternative travel options, check against user’s schedule, and present a proposed plan for confirmation. To support such experiences, we build a proactive service harness based on mobile notifications. Beyond initiating tasks, the harness forms a coherent understanding of real-world situations, generates personalized tasks, executes approved actions, and continuously evolves from user responses and long-term interaction history.

Empirically, Qwen-UI-Agent achieves state-of-the-art performance across a broad range of evaluation settings, covering real-device and benchmark-based mobile use, computer use, browser use, DeepSearch, and GUI grounding. Across these evaluations, Qwen-UI-Agent generally outperforms leading foundation models, including Opus 4.8 (Anthropic, 2026), GPT-5.6 Sol (OpenAI, 2026b), Gemini 3.5 Flash (Google, 2026b), and Seed2.1 Pro (ByteDance Seed, 2026b), as well as specialized GUI agents. We highlight main results below:

• Mobile Use. On MobileWorld-Real, our real-device benchmark comprising more than 400 tasks across over 100 apps, Qwen-UI-Agent achieves a 92.2% success rate. It outperforms frontier closed-source models Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 6.0, 7.5, 6.8, and 3.5 percentage points, respectively. Qwen-UI-Agent further achieves a near-perfect success rate of 97.5% on AndroidDaily, another real-device benchmark. On MobileWorld, Qwen-UI-Agent scores 82.1%, surpassing Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 14.6, 12.0, and 8.9 percentage points.
• Computer Use. On OSWorld-Verified, Qwen-UI-Agent achieves 79.5%, ranking second overall and outperforming GPT-5.5, Gemini 3.1 Pro, and Seed 2.1 Pro. On the more challenging OSWorld-v2, Qwen-UI-Agent obtains the second-best binary success rate (13.9%), the third-best partial-progress score (40.0%). Its partial-progress score exceeds those of MiniMax M3 and Qwen 3.7 Plus by 17.7 and 18.5 percentage points, respectively, while requiring 58.4% and 21.7% fewer steps per task.
• Browser Use and DeepSearch. On WebArena, Qwen-UI-Agent achieves 73.6%, outperforming Claude Opus 4.8, GPT-5.5 and Gemini 3.1 Pro by 1.7, 4.1 and 8.3 points. On DeepSearch, it scores 64.1% on BrowseComp and 75.0% on BrowseComp-ZH, surpassing Qwen3.5-397B-A17B and UI-TARS-2.
• GUI Grounding. Qwen-UI-Agent reaches 81.5% on ScreenSpot-Pro under the zoom-in setting, outperforming Seed 2.1 Pro, GUI-Owl-1.5, and UI-Venus-1.5. It further sets the best results among baseline models on UI-Vision (70.0%), OSWorld-G-Refined (78.5%), MMBench-GUI L2 (92.6%), and ScreenSpot-V2 (97.5%).
• General and Agentic Capabilities. To preserve the model’s broad utility in real-world settings, we retain Qwen-UI-Agent’s general reasoning and agentic capabilities. We evaluate Qwen-UI-Agent on a suite of 13 benchmarks, including MMMU-Pro, MMLU-Pro, Terminal-Bench 2.0, and Claw-Eval. The results show that Qwen-UI-Agent outperforms the Qwen base model on agentic tasks while remaining comparable on general reasoning tasks, and significantly outperforms GUI-specialized models across both categories.

2 Qwen-UI-Agent

Figure 2: An illustrative trajectory of Qwen-UI-Agent for proactive cross-platform task execution. The left panel summarizes the system capabilities underlying the trajectory, including cross-platform execution, a hybrid action space spanning GUI, CLI, and API operations, and environments ranging from large-scale sandboxes to real devices. The right panel illustrates a travel-recovery scenario triggered by a flight-cancellation notification. After identifying the affected tasks and commitments, the agent searches for alternative flights through an API, requests user approval before rebooking via a mobile GUI, updates the affected meeting schedule through desktop GUI and CLI actions, and sends the revised artifact to the relevant recipients.

This section presents the methodology of Qwen-UI-Agent as an integrated system for building advanced foundation GUI agents. The system comprises four components: (1) an environment infrastructure spanning both sandboxed and real-world environments that supports trajectory collection and training across mobile-use, computer-use, browser-use, and deep-search tasks under a unified action space covering GUI, CLI, and batched actions; (2) an automated data flywheel that reduces human intervention and closes the loop from data synthesis to training, evaluation, failure analysis, and targeted data generation for the next iteration; (3) a unified training framework that combines supervised fine-tuning with reinforcement learning; and (4) a harness layer that connects the trained agent to user context and enables proactive services and cross-platform task execution. Figure 2 illustrates a demo trajectory.

2.1 System Overview

Qwen-UI-Agent is designed as a real-world-centric foundation GUI agent that operates across diverse digital environments, including mobile, desktop, web, and information-seeking systems. This section first defines the task scope of Qwen-UI-Agent, then introduces its unified observation and action spaces, and finally summarizes the capability landscape supported by this formulation.

2.1.1 Task Formulation

Qwen-UI-Agent is designed for interactive digital task execution. We define a task as:

\tau=\left(I,\mathcal{E}_{\tau}\right), 

where I is the user instruction and \mathcal{E}_{\tau} is the set of digital environments available to the task. Depending on the task, \mathcal{E}_{\tau} may include mobile devices, web browsers, computer systems, DeepSearch systems, or a combination of them. This formulation covers both tasks completed within a single environment and workflows that span multiple platforms (Xu et al., 2026; Lian et al., 2026).

At decision step t, the agent receives a multichannel observation ot that may combine different forms of environment feedback:

o_{t}=\left(o_{t}^{\mathrm{GUI}},o_{t}^{\mathrm{CLI}},o_{t}^{\mathrm{API}}\right). 

Here, otGUI represents the current screenshot, otCLI contains command execution results, and otAPI contains structured responses from external services, such as search API. A Component is set to empty when the corresponding channel is unavailable or unnecessary for the current task. This multichannel observation allows the agent to jointly reason over visual interface states and programmatic execution feedback (Hu et al., 2026; Li et al., 2026a).

The agent predicts an intermediate reasoning output rt and an executable action output 𝐚t:

\left(r_{t},\mathbf{a}_{t}\right)=\pi_{\theta}\left(I,o_{t},h_{t}\right), 

where h_{t}=\left(o_{1},r_{1},\mathbf{a}_{1},\ldots,o_{t-1},r_{t-1},\mathbf{a}_{t-1}\right) denotes the preceding interaction history.

Notably, one model decision step may produce either a single action or an ordered sequence of actions:

\mathbf{a}_{t}=\left(a_{t}^{(1)},\ldots,a_{t}^{(K_{t})}\right),\qquad a_{t}^{(k)}\in\mathcal{A}_{t}, 

where \mathcal{A}_{t} is the set of actions available in the current environment. When Kt=1, the model performs single-action execution. When Kt>1, it produces a batched action sequence. Actions within a batch are executed consecutively in this decision step, reducing unnecessary inference and observation steps when multiple operations can be completed without additional environment feedback.

2.1.2 Action Space

We design the action space of Qwen-UI-Agent around three requirements: supporting GUI interaction across different platforms, extending execution beyond GUI manipulation, and preserving user control over consequential operations. The GUI action space is defined as the union of actions necessary for mobile, web, and desktop environments, with each environment exposing the subset supported by its native interaction mechanisms. To complement GUI interaction, we introduce cli_command, which enables direct bash command execution, and api_call, which invokes external services with structured arguments. We further include ask_user, allowing the agent to request missing information or obtain explicit confirmation before handling sensitive data, payments, or other consequential operations. Together, these actions provide a unified interface for cross-platform GUI interaction, hybrid GUI+CLI execution, and user-controlled operation in real-world tasks. The full action space is shown in Table 1.

Action	Definition
GUI Actions
click	Clicks at coordinates (x,y).
double_click	Double-clicks at coordinates (x,y).
long_press	Long-presses at coordinates (x,y).
type	Types the specified text content.
open	Opens the specified app.
drag	Drags from start coordinates (x1,y1) to end coordinates (x2,y2).
system_button	Presses a system button, selected from back, home, menu, and enter.
wait	Waits for a specified duration in seconds.
CLI Actions
cli_command	Executes a bash command in the active CLI environment.
API Actions
api_call	Invokes an API with specified arguments.
Interaction and Control Actions
ask_user	Interact with the user to complete the task.
terminate	Ends the task with a final answer and a status of success or failed.

Table 1: Action Space in Qwen-UI-Agent.

2.1.3 Capability Landscape

The unified formulation supports four complementary capabilities of Qwen-UI-Agent centered on real-world task execution. (1) Real-device mobile execution. By building a real-device mobile environment and using it throughout model capability development and evaluation, Qwen-UI-Agent substantially eliminates the simulation-to-reality gap and achieves leading real-world mobile execution performance over both specialized GUI agents and frontier closed-source models. (2) Multi-domain task execution. Qwen-UI-Agent operates across mobile, desktop, web, and DeepSearch environments, moving beyond domain-specific policies toward a foundation GUI agent for heterogeneous digital tasks. (3) Hybrid and efficient interaction. Qwen-UI-Agent’s unified action space enables the model to select and interleave GUI operations with bash-based CLI commands within the same trajectory, while batched actions largely enhance the execution efficiency. (4) Harness-enabled services. A lightweight harness layer extends the core agent beyond isolated task execution by enabling proactive service initiation and allowing a single workflow to span mobile and computer systems.

2.2 Environment Infrastructure

The environment defines the capability boundary of an agent: it determines what the agent can perceive, what actions it can take, and what experience it can acquire through interaction. Therefore, moving GUI agents from benchmark-centric optimization toward real-world utility requires not only stronger models, but also broader and more realistic environments in which agents can learn, act, and adapt. As illustrated in Figure 3, our environment infrastructure is organized around four core components. First, simulated and sandbox environments provide scalable, controllable, and reproducible settings for data generation, agent learning, and evaluation. Second, extending these simulated environments to real devices enables agents to learn and operate under realistic deployment conditions, including dynamic device states, system constraints, and user interaction. Third, a hybrid GUI and CLI action space supports more efficient and adaptive execution by allowing the agent to select the most appropriate interaction modality for each task. Finally, a unified cross-platform environment provides a unified pipeline across mobile-use, computer-use, browser-use, and DeepSearch, while preserving context and task state across platforms. Together, these components make the environment a core part of the agent system, directly shaping the range of tasks the agent can learn and accomplish.

Figure 3: The environment infrastructure of Qwen-UI-Agent. (a) Scalable sandbox environments spanning mobile-use, computer-use, browser-use, and DeepSearch provide controllability and repeatable evaluation for data synthesis and training. (b) A sim-to-real bridge extends agents to real devices with real applications, networks, and account states, supporting user takeover for login, payment, permission, and confirmation. (c) A hybrid GUI+CLI action space interleaves graphical operations with direct command execution. (d) A unified interface standardizes the thinking–action–observation loop across heterogeneous environments.

2.2.1 Scaling Sandbox Environments

Sandbox environments provide controlled and resettable task states for training, evaluation, and ablation. Our suite spans mobile-use, computer-use, browser-use, and DeepSearch tasks. Each domain retains its native applications, environment-specific context, and state-based verification mechanisms. A key strength of our infrastructure is its scalability: it supports up to 10,000 isolated sandbox environments running in parallel, enabling large-scale rollout and agent training.

• Mobile Use. We build our mobile sandbox on the environment provided by MobileWorld, which offers a resettable Android system with 20 applications and supports GUI task execution, agent-user interaction, and MCP tool use (Kong et al., 2026). To address the scaling bottleneck of KVM-based emulators, we rebuilt the MobileWorld environment on redroid (Remote Android Project, 2024), which runs Android as a container on the host kernel without QEMU or nested KVM. This design enables complete device-and-backend environments to be replicated across ordinary container hosts. Its containerized device instances and self-hosted application backends provide controllable initial states and reproducible state transitions. Building on this infrastructure, we construct diverse task-relevant environment contexts, synthesize tasks, and generate state-based verifiers. The resulting sandbox provides a stable and scalable environment for trajectory collection and agent training.
• Computer Use. We build our computer-use sandbox on the Ubuntu virtual-machine environment provided by OSWorld, which supports real desktop applications, operating-system operations, and cross-application workflows (Xie et al., 2024). While the standard OSWorld interaction interface is GUI-based, we extend it with direct bash execution, allowing the agent to interleave GUI operations and CLI commands within the same trajectory. For our newly constructed tasks, we further create task-specific initial environment states and state-based verifiers for some of these tasks. Together, these extensions provide a controllable and scalable environment for collecting trajectories and training agents under both GUI-only and hybrid GUI+CLI interaction.
• Browser Use. Our web environment is a self-contained browser runtime implemented with FastAPI, Playwright (Microsoft, 2020), and Chromium. Each episode runs in a fresh Playwright BrowserContext with isolated cookies, cache, and local storage, and begins from a task-specific page and state. The agent receives rendered screenshots and interacts through browser-native GUI actions. The JavaScript-based verifiers inspect the DOM and persistent application state, allowing alternative action sequences to receive credit when they reach the required state.
• DeepSearch. The DeepSearch environment complements interactive browser control with structured information-access tools. Serper (Serper, 2023) provides ranked search results, while Jina Reader (Jina AI, 2024) converts selected web pages and documents into model-consumable text. The agent can iteratively reformulate queries, select sources, retrieve full content, and synthesize evidence across pages. This environment therefore targets open-ended, multi-source information seeking in which the principal observations are search results and retrieved documents rather than rendered GUI states.

2.2.2 Real-Device Mobile Environment: From Simulated Sandboxes to Real-World Learning

Figure 4: Real-device mobile runtime with closed-loop environment governance. The health-aware scheduler routes each task to an eligible phone, App/account, and display; unhealthy targets remain blacklisted until repair and revalidation. Virtual displays allow one phone to run multiple Apps concurrently. Evidence-based review separates task success, model failure, and environment failure from the complete trajectory, and confirmed environment issues are fed back to the scheduler.

Why Real Devices Matter?

Sandbox environments provide stable and resettable task states, making them essential for large-scale rollout and training. However, they cannot fully capture the evolving application states, permission-related constraints, and or execution-time disruptions such as unexpected pop-ups and network instability. On real devices, application interfaces and content change over time, permissions and CAPTCHA interrupt execution, and network or device failures can occur at any step. The Chinese mobile ecosystem further amplifies these challenges: super Apps, dense interfaces, and frequent pop-ups make it impossible to reproduce such execution paths in a sandbox. To make Qwen-UI-Agent work reliably under these conditions, we build a real-device mobile runtime and use it for model training and evaluation. We further construct MobileWorld-Real, which is introduced in Section 3.2, to systematically evaluate agent performance on real devices.

Real-device Mobile Runtime.

We build a real-device mobile runtime comprising over 100 physical devices and over 150 applications. At this scale, assigning each task a reliable and usable execution device becomes a central systems challenge. As shown in Figure 4, the health-aware scheduler continuously tracks the health and availability of each device, application, account, network connection, and display. When a task enters the queue, it selects an eligible execution target, leases the required resources to the task, and reroutes execution to another target when failures occur. This health-aware orchestration keeps the environment productive despite environment-side failures caused by the inherent instability of real phones, applications, and network connections.

To further scale real-device environment, we leverage the virtual-screen mechanism (Genymobile, 2018) that allows a single phone to host multiple application sessions concurrently on separate displays. The runtime controller binds each virtual display to a corresponding agent session, ensuring that observations are delivered to the correct agent and that its actions are executed on the intended display. This design increases the number of active execution environments without requiring a proportional increase in physical devices. Additionally, real-world tasks often involve either sensitive operations that require explicit user approval or missing information that must be provided by the user. We therefore introduce a dedicated User Agent that can provide missing information, obtain explicit confirmation for data-sensitive or payment-related operations, and hand control to the user when an action requires direct human intervention such as CAPTCHA. After the required input or operation is completed, the task resumes from the updated environment state.

A real-device runtime also needs to distinguish model failures from failures of the execution environment itself. Application restrictions, unavailable services, unstable devices, and network errors may interrupt a trajectory even when the model’s decisions are valid. The scheduler maintains a dynamic blacklist over unhealthy devices, removes them from routing, and restores them only after manual inspection. We employ a VLM-based judge to examine the complete trajectory and to distinguish among task success, model failure, and environment failure. Confirmed environment failures update the Scheduler and maintenance queue, while validated trajectories are retained for training.

2.2.3 Hybrid GUI+CLI Action Space

Real-device environments expand the range of states an agent can encounter, but real-world utility also depends on how effectively the agent can act on them. GUI interaction provides universal access to user-facing applications and visual context, but pure GUI execution may turn structured operations into long sequences of grounding, clicking, and typing. CLI execution offers a direct programmatic access to files, code, system settings, and batch operations, but is less suitable when tasks depend on visual understanding, application-specific interfaces, or services without programmable access. GUI and CLI therefore offer complementary forms of environment access: GUI enables broad application coverage and visually grounded interaction, while CLI provides efficient execution over programmatic state.

We therefore equip Qwen-UI-Agent with the unified GUI+CLI action space defined in Section 2.1.2, and our environments expose both forms of access through a single execution interface. In the computer-use environment, every action is dispatched to a lightweight execution service inside the VM: GUI actions are translated into native input events, while cli_command runs the command in a non-interactive shell, without visually locating and operating a terminal application. The resulting stdout, stderr, and exit status are returned as structured observations alongside the post-action screenshot, allowing the model to condition its next decision jointly on programmatic and visual state.

Reliable CLI execution requires environment-side handling beyond spawning a shell. Command execution is bounded by a timeout budgeted for slow operations such as dependency installation, while failures like non-zero exits and timeouts are returned as error observations rather than aborting the episode, allowing the model to diagnose and recover within the same trajectory. In batched actions, GUI and CLI sub-actions execute in order with their CLI outputs aggregated into a single observation. Executed commands are also recorded into the shell history, so that verifiers inspecting terminal state credit CLI-based solutions as well.

2.2.4 Unified Interface for Heterogeneous Environments

Mobile-use, computer-use, browser-use, and DeepSearch environments expose different resource models, action spaces, observation formats, and verification procedures. We therefore introduce a unified environment interface that standardizes the environment lifecycle and agent-facing inputs and outputs, while delegating execution, reset, and verification to environment-specific adapters. This allows the same agent and training pipeline to operate across heterogeneous environments without imposing identical runtime semantics.

The interface exposes a common asynchronous lifecycle comprising acquire, reset, step, evaluate, tear_down, and release. Each acquired environment is represented as a leased session bound to a backend and, where necessary, a display identifier, which consistently routes screenshots, actions, and evaluations to the same isolated context. Backend adapters implement resource initialization, observation capture, action execution, and evaluation, while task adapters preserve platform-specific reset and verification logic. All environments return a common transition format containing observations, a termination signal, and reward information.

The policy and environment independently declare their native action spaces. When they differ, supported GUI operations are translated through a canonical intermediate representation, with coordinates normalized according to the target platform. The hybrid GUI, CLI, and API actions defined in Section 2.1.2 remain available through native adapters, and their visual or structured results are returned through the same observation stream. This design provides a shared rollout, data-collection, and evaluation pipeline while retaining the native interaction semantics of each platform.

2.3 Agent-Driven Data Flywheel

Continually improving GUI agents requires an iterative, closed-loop process spanning task design, environment construction, verifier development, data curation, model training, evaluation, and error analysis. When humans drive these stages, iteration becomes slow and difficult to scale as the coverage expands across applications, platforms, and capability dimensions. To address this issue, we therefore introduce an AutoResearch-style, agent-driven data flywheel that shifts human involvement from executing each stage to high-level oversight and targeted intervention.

Overall Procedure.

As illustrated in Figure 5, we organize our agent driven data flywheel into two stages: domain capability bootstrapping and an iterative refinement loop. During bootstrapping, we use strong foundation models to analyze the domain knowledge and capabilities required for the various domain covering mobile, desktop and web. Guided by this analysis, the agents generate an initial pool of tasks and corresponding environment contexts, from which candidate trajectories are collected. Following MAI-UI (Zhou et al., 2025a), we first perform multiple rounds of rejection sampling and aggregate the accepted trajectories into a unified SFT corpus. This process yields a strong initial policy with broad coverage of the target domain. Building on this policy, our agent-driven failure analysis identifies the model’s remaining weaknesses and maps them to targeted goals for the next cycle. These goals guide the generation of new tasks, environment configurations, and, where applicable, task-specific verifiers. The accepted trajectories are then incorporated into the training corpus to further improve the model, thereby closing the loop and initiating the next iteration. Across both the initial policy construction and iterative improvement stages, strong foundation models generate tasks and environment configurations, synthesize task-specific verifiers, evaluate trajectories, analyze failures, and plan the optimization and execution of subsequent iterations. Human involvement primarily focuses on designing and implementing the overall system, overseeing its operation, and intervening or revising the workflow when necessary. We next describe four design choices that make this agent-driven data flywheel scalable and effective.

Figure 5: The data flywheel of Qwen-UI-Agent. Domain capability bootstrapping initializes training, iterative refinement loop identifies capability weaknesses and generate targeted tasks, and the resulting data improve the next training iteration.

Knowledge- and Capability-Aware Task Synthesis.

Training tasks define the learning signals available to the model and therefore shape the knowledge and behaviors it can acquire. We organize task synthesis along two conceptually distinct dimensions: knowledge coverage, which determines what the agent needs to know, and capability demand, which determines how the agent must reason and act to complete the task. Knowledge coverage specifies the domain-specific operational knowledge, including application functions, interface conventions, tool usage, and common workflows. Capability demand captures the transferable reasoning and interaction abilities, such as long-horizon state tracking, precise constraint following, numerical and temporal reasoning, and error recovery. For each target domain, we use strong foundation models to construct an initial hierarchical function tree for knowledge coverage and a capability profile for capability demand. We then compile them into a reusable task-synthesis skill containing composition rules, difficulty controls, environment prerequisites, and representative examples. Conditioned on the objective of the current iteration, this skill lets the agent automatically generate task toward the required knowledge and capabilities.

Agent-Driven Environment and Verifier Synthesis.

Task synthesis determines the capabilities that a training instance should elicit, but interactive training additionally requires a reproducible environment in which the task is feasible and executable. Given a synthesized task, our agents use Environment State Synthesis, detailed in Section 2.4.3, to construct the application state, files, records, accounts, and cross-application context needed for execution. For tasks with reliably checkable outcomes, agents further synthesize state-based verifiers that inspect the resulting application, file, database, or system state, yielding executable task–verifier pairs .

Step-Level Judging for Scalable SFT and Executable Verification for Online RL.

Executable verifiers provide high-precision outcome signals, but constructing and validating one is costly. Even with stronger foundation models, we find that synthesizing and validating executable verifiers remains time-consuming, whereas the accuracy of model-based judges improves alongside model development. We therefore extend the step-level trajectory judge used in MAI-UI (Zhou et al., 2025a) and apply a VLM-based judge to mobile, web, and computer-use trajectories, including hybrid GUI+CLI interactions. Given the instruction, trajectory, and visual or structured observations, the judge extracts three forms of useful supervision: maximal contiguous steps that correctly advance the task, the first step that initiates each reflection or exploration phase, and recovery segments that return execution from an erroneous state to a valid path. Retaining only the first reflection or exploration step captures the useful decision to reconsider without over-keeping a potentially noisy branch. This fine-grained procedure recovers effective training signal from both successful and failed trajectories. Empirically, we find that step-level data filtering yields SFT performance comparable to or better than training on complete trajectories selected by executable verifiers. We therefore use fine-grained VLM judgment to scale process supervision for SFT, while constructing executable verifiers for the high-precision outcome signals required by online RL.

Failure Analysis Driven Iteration.

We construct a broad and diagnostic evaluation suite for each domain to track model performance and define the optimization objectives of the next iteration. For each failed task, an analysis agent examines the complete trajectory and distinguishes model failures from environment, task, or verifier failures. Model failures are then mapped to a structured cause, such as missing application knowledge, constraint violations, state-tracking errors, or insufficient verification. The system aggregates these task-level diagnoses to identify the most prevalent and consequential weaknesses of the current policy and converts them into a prioritized set of optimization objectives. Conditioned on these objectives and the task-synthesis skill, agents generate targeted tasks and corresponding environment setups and, where applicable, task-specific verifiers for the next cycle. Non-model issues are routed to environment maintenance or task and verifier revision, where agents diagnose and address the identified problems accordingly.

2.4 Training

2.4.1 Supervised Fine-Tuning (SFT)

Qwen-UI-Agent is designed to support task execution across mobile, computer-use, web, and DeepSearch settings. This scope poses two coupled challenges for SFT: jointly acquiring strong capabilities across all target domains within a single model, and preserving the general reasoning and agentic competence needed for robust real-world task completion.

Domain-Conditioned Expert Training.

Across mobile, desktop, and web, each domain places different demands on the agent. Mobile tasks require reliable execution in real-device environments, where interfaces are app-specific, workflows are frequently interrupted by pop-ups, and safety-sensitive actions must remain under user control. Desktop tasks often combine GUI control with Bash-based file and system operations. Web tasks require interaction with dynamic webpages, while DeepSearch tasks require the agent to retrieve, verify, and synthesize evidence through Search APIs.

We train a specialized expert for each domain and then consolidate their capabilities. Each expert is trained mainly on data from its target domain, together with a controlled mixture of data from other domains. This cross-domain mixture helps each expert retain capabilities that transfer across domains and reduces overfitting to its target domain. Practical deployment requires a single model that can operate across all supported settings. We therefore merge the domain-expert checkpoints into a unified model. Model merging combines their complementary strengths within a single checkpoint, preserving the capabilities acquired by the individual experts with little additional training overhead.

Preserving General and Agentic Capabilities with In-Distribution Data.

General reasoning and agentic capabilities are essential for real-world use, where agents need to respond to diverse and open-ended user requests. Some requests may even be unrelated to GUI tasks, yet the model should still respond appropriately. Even within GUI tasks, real-world deployment exposes the agent to out-of-distribution (OOD) states and objectives that may require knowledge, reasoning, instruction following, retrieval, coding, or tool use not anticipated during training. We therefore seek to preserve and, where possible, strengthen the general reasoning and agentic capabilities inherited from the base model, providing an important foundation for handling unfamiliar situations and maintaining a consistent user experience.

We find that the most effective strategy is to use in-distribution data that reinforces successful behaviors that the starting model can produce. We construct a broad query pool covering general question answering, mathematics, coding, visual understanding, search, and agentic tool use, sample responses from the starting model, and retain verified correct examples after format normalization. A small proportion of these examples is mixed with GUI trajectories during SFT. In our experiments, examples that the starting model is capable of solving are substantially more effective for capability preservation than challenging examples on which it fails. The former rehearse established capabilities using targets aligned with the model’s own response distribution, whereas the latter shift training toward capability acquisition and can introduce competing optimization signals.

Sliding-Window Training for Efficient Long-Trajectory SFT.

Long-trajectory SFT becomes expensive when every action is packed with its visual and textual context as a separate training sample, causing highly overlapping context to be processed repeatedly across adjacent actions. We therefore divide each trajectory into windows of n=5 consecutive steps and advance each window by n−1=4 steps, leaving a one-step overlap between adjacent windows. This overlap ensures that the first newly supervised action in each subsequent window can condition on at least two screenshots. In each subsequent window, the shared boundary step remains in the input context, but its loss is masked because it has already been supervised in the preceding window. Each full subsequent window therefore supervises four new actions. Screenshots before the current window are omitt...