# Computer Use

Use when a task needs an app on the user's own screen, a native desktop application...

- Repository: https://github.com/ai4s-research/open-science (2069 stars)
- Category: Documents
- License: see repository
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ai4s-research/open-science --skill computer-use -g -a claude-code -y`
Codex: `npx skills add ai4s-research/open-science --skill computer-use -g -a codex -y`
Cursor: `npx skills add ai4s-research/open-science --skill computer-use -g -a cursor -y`
Gemini CLI: `npx skills add ai4s-research/open-science --skill computer-use -g -a gemini-cli -y`

## Description

Use when a task needs an app on the user's own screen, a native desktop application (instrument control software, Origin, GraphPad, ImageJ, a reference manager, a spreadsheet), or a desktop browser window that needs window-level control. Read before the first `computer` call. Covers the observe-act-observe loop, why element indexes go stale, which action to prefer for which control, what "unverified" means and why it must never be reported as success, and the recovery for each error code. Not for page-level web automation (use the browser connector) and not for anything that can be done from the shell.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Computer use

`computer` reads an app's **accessibility tree** and acts on the numbered
elements in it. The tree is the interface; a screenshot is confirmation. This
design, and most of the rules below, come from
[Orca](https://github.com/stablyai/orca) (MIT).

## Before anything else

Use this only when there is no better route. A file is read with `read`, a
computation is run in the shell, a web page is driven by the browser connector.
Reach for `computer` when the capability lives in a desktop app and nowhere else.

The user sees a permission prompt on the first call of a session. Say what you
intend to do before you make it.

**Never** push, submit a form, send a message, buy, delete, or change an account
setting unless the user asked for that exact thing. If an app holds content the
user did not ask about, read only what the task needs.

## The loop

```
computer(action: "list_apps")
computer(action: "get_state", app: "com.spotify.client")
computer(action: "click", app: "com.spotify.client", elementIndex: 42)
computer(action: "get_state", app: "com.spotify.client")
```

Observe, act once, observe again. Every action already returns the new state, so
the reply to an action is also the next thing you read.

## Element indexes go stale

The numbers in the tree are valid for **that** snapshot of **that** window.
Navigation, scrolling, a focus change, a dialog, an app re-render, or simply a
few seconds passing can invalidate them.

- Re-read the tree after anything that changes the UI.
- Indexes are **sparse**, noisy sections are omitted. Never infer a valid index
  from the element count, and never guess one that is not printed.
- One window at a time: once you pick `windowId` (or `windowIndex`), pass the
  same one to every later call until you mean to change window.

## Picking the app

The rest: https://github.com/ai4s-research/open-science/blob/HEAD/runtime/skills/core/computer-use/SKILL.md