Mac Use

Control any macOS GUI application through a screenshot → pick element → click → verify loop.

Setup

Platform: macOS only (requires Apple Vision framework for OCR)

System binaries (pre-installed on macOS):

• python3, via Homebrew (brew install python)
• screencapture, built-in macOS utility

Python packages, install from the skill directory:

pip3 install --break-system-packages -r {baseDir}/requirements.txt

How It Works

The screenshot command captures a window, uses Apple Vision OCR to detect all text elements, draws numbered annotations on the image, and returns both:

1. Annotated image at /tmp/mac_use.png, numbered green boxes around each detected text
2. Element list in JSON, [{num: 1, text: "Submit", at: [500, 200]}, {num: 2, text: "Cancel", at: [600, 200]}, ...] where at is the center point [x, y] on the 1000x1000 canvas (origin at top-left)

You receive both by calling Bash (gets JSON with element list) and then Read on /tmp/mac_use.png (gets the visual). Always do both so you can cross-reference the numbers with what you see.

Quick Reference

# List all visible windows
python3 {baseDir}/scripts/mac_use.py list

# Screenshot + annotate (returns image + numbered element list)
python3 {baseDir}/scripts/mac_use.py screenshot <app> [--id N]

# Click element by number (primary click method)
python3 {baseDir}/scripts/mac_use.py clicknum <N>

# Click at canvas coordinates (fallback for unlabeled icons)
python3 {baseDir}/scripts/mac_use.py click --app <app> [--id N] <x> <y>

# Scroll inside a window
python3 {baseDir}/scripts/mac_use.py scroll --app <app> [--id N] <direction> <amount>

# Type text (uses clipboard paste, supports all languages)
python3 {baseDir}/scripts/mac_use.py type [--app <app>] "text here"

# Press key or combo
python3 {baseDir}/scripts/mac_use.py key [--app <app>] <combo>

Workflow