# Mac Use

Control macOS GUI apps visually, take screenshots, click, scroll, type.

- Repository: https://github.com/polyuiislab/infiAgent (1187 stars)
- Category: Engineering
- License: GPL-3.0
- Last updated: 2026-07-11
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add polyuiislab/infiAgent --skill mac-use -g -a claude-code -y`
Codex: `npx skills add polyuiislab/infiAgent --skill mac-use -g -a codex -y`
Cursor: `npx skills add polyuiislab/infiAgent --skill mac-use -g -a cursor -y`
Gemini CLI: `npx skills add polyuiislab/infiAgent --skill mac-use -g -a gemini-cli -y`

## Description

Control macOS GUI apps visually, take screenshots, click, scroll, type. Use when the user asks to interact with any Mac desktop application's graphical interface.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Mac Use

Control any macOS GUI application through a **screenshot → pick element → click → verify** loop.

## Setup

**Platform**: macOS only (requires Apple Vision framework for OCR)

**System binaries** (pre-installed on macOS):
- `python3`, via Homebrew (`brew install python`)
- `screencapture`, built-in macOS utility

**Python packages**, install from the skill directory:
```bash
pip3 install --break-system-packages -r {baseDir}/requirements.txt
```

## How It Works

The `screenshot` command captures a window, uses **Apple Vision OCR** to detect all text elements, draws numbered annotations on the image, and returns both:
1. **Annotated image** at `/tmp/mac_use.png`, numbered green boxes around each detected text
2. **Element list** in JSON, `[{num: 1, text: "Submit", at: [500, 200]}, {num: 2, text: "Cancel", at: [600, 200]}, ...]` where `at` is the center point `[x, y]` on the 1000x1000 canvas (origin at top-left)

You receive both by calling Bash (gets JSON with element list) and then Read on `/tmp/mac_use.png` (gets the visual). **Always do both** so you can cross-reference the numbers with what you see.

## Quick Reference

```bash
# List all visible windows
python3 {baseDir}/scripts/mac_use.py list

# Screenshot + annotate (returns image + numbered element list)
python3 {baseDir}/scripts/mac_use.py screenshot <app> [--id N]

# Click element by number (primary click method)
python3 {baseDir}/scripts/mac_use.py clicknum <N>

# Click at canvas coordinates (fallback for unlabeled icons)
python3 {baseDir}/scripts/mac_use.py click --app <app> [--id N] <x> <y>

# Scroll inside a window
python3 {baseDir}/scripts/mac_use.py scroll --app <app> [--id N] <direction> <amount>

# Type text (uses clipboard paste, supports all languages)
python3 {baseDir}/scripts/mac_use.py type [--app <app>] "text here"

# Press key or combo
python3 {baseDir}/scripts/mac_use.py key [--app <app>] <combo>
```

## Workflow

The rest: https://github.com/polyuiislab/infiAgent/blob/HEAD/skills/mac-use/SKILL.md