# 01 Auto Arena

Automatically evaluate and compare multiple AI models or agents without pre-existing...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -g -a gemini-cli -y`

## Description

Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Auto Arena Skill

End-to-end automated model comparison using the OpenJudge `AutoArenaPipeline`:

1. **Generate queries**, LLM creates diverse test queries from task description
2. **Collect responses**, query all target endpoints concurrently
3. **Generate rubrics**, LLM produces evaluation criteria from task + sample queries
4. **Pairwise evaluation**, judge model compares every model pair (with position-bias swap)
5. **Analyze & rank**, compute win rates, win matrix, and rankings
6. **Report & charts**, Markdown report + win-rate bar chart + optional matrix heatmap

## Prerequisites

```bash
# Install OpenJudge
pip install py-openjudge

# Extra dependency for auto_arena (chart generation)
pip install matplotlib
```

## Gather from user before running

| Info | Required? | Notes |
|------|-----------|-------|
| Task description | Yes | What the models/agents should do (set in config YAML) |
| Target endpoints | Yes | At least 2 OpenAI-compatible endpoints to compare |
| Judge endpoint | Yes | Strong model for pairwise evaluation (e.g. `gpt-4`, `qwen-max`) |
| API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. |
| Number of queries | No | Default: `20` |
| Seed queries | No | Example queries to guide generation style |
| System prompts | No | Per-endpoint system prompts |
| Output directory | No | Default: `./evaluation_results` |
| Report language | No | `"zh"` (default) or `"en"` |

## Quick start

### CLI

```bash
# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save

# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
  --queries_file queries.json --save

# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save

# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save
```

### Python API

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/arena-eval/01-auto-arena/SKILL.md