# Agent Eval

Benchmarks CodeGraph retrieval by comparing agent runs with and without it.

- Repository: https://github.com/colbymchenry/codegraph (76154 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add colbymchenry/codegraph --skill agent-eval -g -a claude-code -y`
Codex: `npx skills add colbymchenry/codegraph --skill agent-eval -g -a codex -y`
Cursor: `npx skills add colbymchenry/codegraph --skill agent-eval -g -a cursor -y`
Gemini CLI: `npx skills add colbymchenry/codegraph --skill agent-eval -g -a gemini-cli -y`

## Description

Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# CodeGraph Quality Audit

Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen
codegraph version on a chosen real-world repo. Drives the harness in
`scripts/agent-eval/`.

## Prerequisites
- `tmux` 3+, a logged-in `claude` CLI, `node`, `git` (macOS/Linux).
- Run from the codegraph repo root.

## Workflow

Copy this checklist:
```
- [ ] 1. Pick version (local or npm)
- [ ] 2. Pick language
- [ ] 3. Pick repo by size
- [ ] 4. Pick harness (headless / tmux / both)
- [ ] 5. Run audit.sh in the background
- [ ] 6. Report results
```

**Step 1, version.** Ask with `AskUserQuestion`: which codegraph version to test.
Offer "Local dev build" and "Latest published"; the free-text "Other" lets the
user type a specific version (e.g. `0.7.10`). Map the answer to a VERSION token:
- "Local dev build" → `local`
- "Latest published" → `latest`
- a typed version → that string (e.g. `0.7.10`)

**Step 2, language.** Read `.claude/skills/agent-eval/corpus.json`. Ask with
`AskUserQuestion` which language to test, listing the languages that have entries.

**Step 3, repo.** From the chosen language's entries, ask which repo. Label each
option with its size and file count, e.g. `excalidraw, Medium (~600 files)`.
Each entry carries the `repo` URL and a representative `question`.

**Step 4, harness.** Ask with `AskUserQuestion` which harness to run, and map
the answer to a MODE token:
- "Headless" → `headless`, `claude -p` with stream-json: exact tokens/cost and a
  clean tool sequence (2 runs, fast, no TTY).
- "Interactive (tmux)" → `tmux`, drives the real Claude TUI in tmux: faithful
  Explore-subagent behavior, metrics from session logs (2 runs, slower).
- "Both" → `all`, headless + interactive (4 runs).

**Step 5, run.** Launch in the background (sets the version, clones if missing,
wipes + re-indexes, runs the chosen arms, several minutes):
```bash
scripts/agent-eval/audit.sh <VERSION> <repo-name> <repo-url> "<question>" <MODE>
```

The rest: https://github.com/colbymchenry/codegraph/blob/HEAD/.claude/skills/agent-eval/SKILL.md