Computer Use
Use when a task needs an app on the user's own screen, a native desktop application...
Install
npx skills add ai4s-research/open-science --skill computer-use -g -a claude-code -y- Repository
- ai4s-research/open-science2.1k
- Category
- Documents
- Updated
- 3d ago
- License
- See repository
- Context cost
- Mediumabout 2,069 tokens
- Runs scripts
- No, instructions only
What it does
Use when a task needs an app on the user's own screen, a native desktop application (instrument control software, Origin, GraphPad, ImageJ, a reference manager, a spreadsheet), or a desktop browser window that needs window-level control. Read before the first `computer` call. Covers the observe-act-observe loop, why element indexes go stale, which action to prefer for which control, what "unverified" means and why it must never be reported as success, and the recovery for each error code. Not for page-level web automation (use the browser connector) and not for anything that can be done from the shell.
Computer use
computer reads an app's accessibility tree and acts on the numbered
elements in it. The tree is the interface; a screenshot is confirmation. This
design, and most of the rules below, come from
Orca (MIT).
Before anything else
Use this only when there is no better route. A file is read with read, a
computation is run in the shell, a web page is driven by the browser connector.
Reach for computer when the capability lives in a desktop app and nowhere else.
The user sees a permission prompt on the first call of a session. Say what you intend to do before you make it.
Never push, submit a form, send a message, buy, delete, or change an account setting unless the user asked for that exact thing. If an app holds content the user did not ask about, read only what the task needs.
The loop
computer(action: "list_apps")
computer(action: "get_state", app: "com.spotify.client")
computer(action: "click", app: "com.spotify.client", elementIndex: 42)
computer(action: "get_state", app: "com.spotify.client")
Observe, act once, observe again. Every action already returns the new state, so the reply to an action is also the next thing you read.
Element indexes go stale
The numbers in the tree are valid for that snapshot of that window. Navigation, scrolling, a focus change, a dialog, an app re-render, or simply a few seconds passing can invalidate them.
- Re-read the tree after anything that changes the UI.
- Indexes are sparse, noisy sections are omitted. Never infer a valid index from the element count, and never guess one that is not printed.
- One window at a time: once you pick
windowId(orwindowIndex), pass the same one to every later call until you mean to change window.
