jev-desktop

This skill requires a clone of the agent-desktop repository (the scripts live in scripts/jev/ and ship in no installed package) and a TYPESAFE_API_KEY.

Two entry points over the same screen reading. run.mjs takes a goal and drives until it is met. act.mjs takes one step and hands the plan back to you. Neither puts the tree in your context.

The loop

node scripts/jev/run.mjs --app Finder "open the Applications folder"
node scripts/jev/run.mjs --app TextEdit --cursor --text "A line to write." "write the supplied sentence into the document"

Each turn reads the screen, asks for one operation and a target for that operation in the same request, carries it out, and reads again. It prints one JSON line per turn and one when it stops (shown trimmed).

{"turn":{"step":1,"operation":"CLICK","target":"treeitem \"Applications\"",
         "confidence":1,"ok":true,"delivery":"delivered_verified","changed":true}}
{"stop":"done","confidence":0.96}

The operations are CLICK, TYPE_TEXT, CHECK, UNCHECK, EXPAND, COLLAPSE, SCROLL, DRILL, WIDEN, WAIT, DONE and BLOCKED. Only the ones something on screen can receive are offered: DRILL for a region that reports it holds more, WIDEN only after a DRILL. SCROLL always scrolls down. delivery is waited for WAIT and looked for DRILL and WIDEN.

The target is asked once per operation. click_target is chosen from the elements that advertise Click, type_text_target from those that advertise SetValue, and so on. Only the head matching the chosen operation is read, so a target can never be incompatible with the verb that acts on it.

Looking is an operation. A window is read with --skeleton first, so a dense document costs the same first look as a small panel. A region that was cut off reports how much it holds, and DRILL pins it as the root for later turns. WIDEN gives the whole window back.