agent-desktop

agent-desktop is a CLI that lets an AI agent observe and operate desktop apps through the OS accessibility tree. It is a tool, not an agent. Every command prints one JSON envelope (ok, data or error). The observe, act and verify loop runs in you.

Shell syntax

Examples use a POSIX shell. In PowerShell, quote every ref ('@s8f3k2p9:e1'), because a bare @token is the splatting operator. Use $env:NAME = 'value' instead of export, and | ConvertFrom-Json instead of $(...) and jq. The Windows skill has the PowerShell forms.

First run

1. Run agent-desktop status. It reports the platform, permissions and supported_surfaces.
2. Run agent-desktop skills get platform. It loads the skill for the OS the binary runs on (macOS, Windows or Linux).
3. For flags and syntax, run agent-desktop <command> --help. This skill does not repeat flag lists.

The loop

Use find when you know the role or exact name. Otherwise map the app first and drill into one region.

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region

• A small app needs no skeleton. Take a full snapshot -i.
• Never skeleton a surface (--surface menu, sheet, alert). A surface is already focused.
• An action that opens a menu, sheet or alert lists it in data.surfaces. Target that overlay next.
• After an async step, use wait (see commands-system.md). A fixed sleep is a race.

Refs