Evaluate a skill

A skill edit is only as good as the reports it produces. This workflow runs the candidate skill in sessions that know nothing except the skill file, a one-line business brief, and a locked-down local MCP endpoint, then compares what came back.

Prerequisites

• Local server in no-auth mode: AUTH_MODE=local_noauth in .env.local, then pnpm db:migrate:local and pnpm dev (or pnpm dev:agents). Pass the URL it prints, with /mcp appended, as --endpoint. Default local D1, never remote bindings or remote Postgres. Keep it private; it has no login.
• codex CLI on the path. Runs use its configured default model at high reasoning; record the resolved model from the events log if you need to name it.
• Research credits: the local server still calls DataForSEO. An audit run makes 20 to 40 tool calls. Do not run more sessions than the comparison needs.

Run

node .agents/skills/evaluate-skill/scripts/run.mjs \
  --skill seo-audit \
  --site example.com --site holdout.example \
  --brief "Evaluation brief: Acme at example.com sells <what>. The goal is relevant organic visitors who could become customers. No first-party analytics are connected." \
  --brief "Evaluation brief: <the holdout business>. ..." \
  --endpoint http://<local-dev-host>/mcp

Runs take about 10 to 14 minutes and execute in parallel. Launch the script detached, because a tool-call timeout that kills the shell kills the sessions too:

(nohup node .agents/skills/evaluate-skill/scripts/run.mjs ... > .logs/eval.log 2>&1 &)

What the script does, per site: