agent-desktop doesn't take screenshots. It reads the OS accessibility tree and returns it as JSON, with a ref attached to every actionable element. An agent clicks or types via that ref, then re-reads the UI to check what happened.

That's the whole idea. The original is vercel-labs/agent-desktop (14547 stars, written in Rust, Apache-2.0). A Chinese translation now exists at yangshun2005/agent-desktop-cn, published today. It has 0 stars — it's brand new, so treat it as a reading aid, not a community-validated fork.

This article covers what the tool does, how the ref model works, where it breaks, and what the Chinese version actually adds.

What the tool is, and what it deliberately isn't

The official one-liner is "Computer use CLI for AI agents." Two things follow from that phrasing:

  • It ships as a CLI and a C library. Not a framework, not an SDK with an agent loop inside.
  • No model is bundled. The observe → act → observe loop lives in whatever agent calls it.

That second point matters more than it looks. A lot of "computer use" tooling hides a planner inside, which makes failures hard to attribute. Here, if the agent clicks the wrong thing, the agent chose the wrong ref. The tool just reported the tree.

Supported platforms: macOS, Windows, Linux. Architectures: x64 and ARM64.

The ref model

A snapshot returns an envelope. Trimmed example from the README:

{ "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 59, "complete": true, "tree": { "role": "window", "name": "Downloads", "children": [ { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] } ] } } }

Two details in that payload are load-bearing.

The ref carries its snapshot id. @s8f3k2p9:e8, not @e8. Write the bare form and you get INVALID_ARGS. This is intentional: a ref without a snapshot id has no way to tell you whether the UI it referred to still exists.

complete is a field, not an assumption. If the tree was truncated, the envelope says so. An agent that ignores it may act on a partial view and conclude the element is missing.

The loop, in shell form:

agent-desktop snapshot --app Finder -i # interactive elements only, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify

The -i flag filters to interactive elements. On a busy window the full tree can be large; -i is usually what you want for the act step.

Installation paths

npm is the shortest route, and needs Node 18+:

npm install -g agent-desktop agent-desktop status # platform, permissions, supported surfaces agent-desktop permissions --guide # macOS: walks you through Accessibility authorization

Bun, Yarn and pnpm install the same package. It bundles native binaries for x64 and ARM64 plus platform helpers.

Alternatives:

  • Homebrew on macOS and Linux: brew install agent-desktop. No official Scoop package; the repo no longer maintains a custom Homebrew tap or Scoop bucket.
  • GitHub Releases: download the matching native archive and place the helper next to the executable, if you want to skip the JavaScript launcher.
  • From source: requires Rust. Cargo.toml declares rust-version = "1.89" and edition 2024.

On macOS, nothing works until Accessibility permission is granted. agent-desktop permissions --guide is the guided path; skipping it means snapshots come back empty and you'll spend an hour debugging the wrong layer.

Failure modes worth knowing before you write the loop

These are the ones that cost time.

Stale refs. If the UI changed between snapshot and action, the command returns STALE_REF or AMBIGUOUS_TARGET. The fix is always the same: re-snapshot, use the new ref. Don't retry the old one. An agent that retries blindly will burn turns.

PowerShell quoting. On Windows PowerShell, @token is the splatting operator. Every ref must be quoted:

agent-desktop click '@s8f3k2p9:e8'

Unquoted, the shell eats it before the CLI sees it.

Blocked postinstall. If npm blocks the postinstall script, every invocation pays roughly 125ms extra. Reinstall with --allow-scripts=agent-desktop. For project-local installs there's a separate npm install-scripts approve flow.

Privacy. Snapshots, clipboard reads, screenshots, command output and debug artifacts can all contain private desktop content. The README says this plainly, and it's not boilerplate. A snapshot of a mail client is a snapshot of your inbox. Don't put these in public logs or upload them to a shared trace viewer.

What the Chinese version adds

The translated repo is structured in two layers, and the split is the useful part.

The top section — 「中文快速上手」 — is explicitly labeled as supplementary, not part of the original. It answers the questions a Chinese-speaking developer hits in the first ten minutes:

  • What problem does this solve, in one paragraph
  • The shortest install-to-first-snapshot path
  • Environment requirements (Node 18+, x64/ARM64, the Homebrew and source-build routes)
  • A "common pitfalls" list: the snapshot-id requirement, PowerShell quoting, the postinstall penalty, the privacy warning, macOS authorization

Below that, the README body is a translation of the original documentation, with a stated goal of staying faithful to the source. The original English README is kept as README_en.md for comparison. Translation errors are handled by Issue or PR, which is the right call — a translated technical doc that can't be diffed against its source is worse than no translation.

The value here is narrow but real. The English README is not long, but it's dense with platform-specific caveats — the PowerShell splatting issue, the STALE_REF semantics, the macOS permission gate. Those are exactly the sentences a non-native reader skims and then loses an afternoon to. Having them in Chinese, with the pitfalls pulled forward into a checklist, shortens the path to a working snapshot.

One caveat on the translation date: the header lists 2026-11-05. The upstream project moves; if you're reading this much later, check README_en.md for drift.

Where this fits

If you're building an agent that needs to drive a real desktop app, the accessibility-tree approach has a clear advantage over pixel matching: elements come back with roles, names and available actions, so the agent reasons over semantics instead of coordinates. It also has a clear cost: you're dependent on how well each app implements accessibility. Apps with poor trees will give you poor snapshots, and no amount of prompt engineering fixes that.

The tool is honest about its scope. It reads, it acts, it reports. The loop is yours.

Repo: yangshun2005/agent-desktop-cn — Chinese translation, published today, 0 stars. Upstream: vercel-labs/agent-desktop, 14547 stars, Rust, Apache-2.0.

Chinese documentation: https://github.com/yangshun2005/agent-desktop-cn

I'm an independent enterprise AI delivery consultant — private LLM deployment, RAG, agent workflows, AIOps. If your team is stuck somewhere in that stack, my inbox is open.