01 Auto Arena
Automatically evaluate and compare multiple AI models or agents without pre-existing...
Install
npx skills add agentscope-ai/OpenJudge --skill 01-auto-arena -g -a claude-code -y- Repository
- agentscope-ai/OpenJudge1k
- Category
- Testing
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,316 tokens
- Runs scripts
- No, instructions only
What it does
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.
Auto Arena Skill
End-to-end automated model comparison using the OpenJudge AutoArenaPipeline:
- Generate queries, LLM creates diverse test queries from task description
- Collect responses, query all target endpoints concurrently
- Generate rubrics, LLM produces evaluation criteria from task + sample queries
- Pairwise evaluation, judge model compares every model pair (with position-bias swap)
- Analyze & rank, compute win rates, win matrix, and rankings
- Report & charts, Markdown report + win-rate bar chart + optional matrix heatmap
Prerequisites
# Install OpenJudge
pip install py-openjudge
# Extra dependency for auto_arena (chart generation)
pip install matplotlib
Gather from user before running
| Info | Required? | Notes |
|---|---|---|
| Task description | Yes | What the models/agents should do (set in config YAML) |
| Target endpoints | Yes | At least 2 OpenAI-compatible endpoints to compare |
| Judge endpoint | Yes | Strong model for pairwise evaluation (e.g. gpt-4, qwen-max) |
| API keys | Yes | Env vars: OPENAI_API_KEY, DASHSCOPE_API_KEY, etc. |
| Number of queries | No | Default: 20 |
| Seed queries | No | Example queries to guide generation style |
| System prompts | No | Per-endpoint system prompts |
| Output directory | No | Default: ./evaluation_results |
| Report language | No | "zh" (default) or "en" |
Quick start
CLI
# Run evaluation
python -m cookbooks.auto_arena --config config.yaml --save
# Use pre-generated queries
python -m cookbooks.auto_arena --config config.yaml \
--queries_file queries.json --save
# Start fresh, ignore checkpoint
python -m cookbooks.auto_arena --config config.yaml --fresh --save
# Re-run only pairwise evaluation with new judge model
# (keeps queries, responses, and rubrics)
python -m cookbooks.auto_arena --config config.yaml --rerun-judge --save
