03 Ref Hallucination Arena
Benchmark LLM reference recommendation capabilities by verifying every cited paper...
Install
npx skills add agentscope-ai/OpenJudge --skill 03-ref-hallucination-arena -g -a claude-code -y- Repository
- agentscope-ai/OpenJudge1k
- Category
- AI and agents
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,290 tokens
- Runs scripts
- No, instructions only
What it does
Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.
Reference Hallucination Arena Skill
Evaluate how accurately LLMs recommend real academic references using the
OpenJudge RefArenaPipeline:
- Load queries, from JSON/JSONL dataset
- Collect responses, BibTeX-formatted references from target models
- Extract references, parse BibTeX entries from model output
- Verify references, cross-check against Crossref / PubMed / arXiv / DBLP
- Score & rank, compute verification rate, per-field accuracy, discipline breakdown
- Generate report, Markdown report + visualization charts
Prerequisites
# Install OpenJudge
pip install py-openjudge
# Extra dependency for ref_hallucination_arena (chart generation)
pip install matplotlib
Gather from user before running
| Info | Required? | Notes |
|---|---|---|
| Config YAML path | Yes | Defines endpoints, dataset, verification settings |
| Dataset path | Yes | JSON/JSONL file with queries (can be set in config) |
| API keys | Yes | Env vars: OPENAI_API_KEY, DASHSCOPE_API_KEY, etc. |
| CrossRef email | No | Improves API rate limits for verification |
| PubMed API key | No | Improves PubMed rate limits |
| Output directory | No | Default: ./evaluation_results/ref_hallucination_arena |
| Report language | No | "en" (default) or "zh" |
| Tavily API key | No | Required only if using tool-augmented mode |
Quick start
CLI
# Run evaluation with config file
python -m cookbooks.ref_hallucination_arena --config config.yaml --save
# Resume from checkpoint (default behavior)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save
# Start fresh, ignore checkpoint
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save
# Override output directory
python -m cookbooks.ref_hallucination_arena --config config.yaml \
--output_dir ./my_results --save
