# 03 Ref Hallucination Arena

Benchmark LLM reference recommendation capabilities by verifying every cited paper...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: AI and agents
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill 03-ref-hallucination-arena -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill 03-ref-hallucination-arena -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill 03-ref-hallucination-arena -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill 03-ref-hallucination-arena -g -a gemini-cli -y`

## Description

Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Reference Hallucination Arena Skill

Evaluate how accurately LLMs recommend real academic references using the
OpenJudge `RefArenaPipeline`:

1. **Load queries**, from JSON/JSONL dataset
2. **Collect responses**, BibTeX-formatted references from target models
3. **Extract references**, parse BibTeX entries from model output
4. **Verify references**, cross-check against Crossref / PubMed / arXiv / DBLP
5. **Score & rank**, compute verification rate, per-field accuracy, discipline breakdown
6. **Generate report**, Markdown report + visualization charts

## Prerequisites

```bash
# Install OpenJudge
pip install py-openjudge

# Extra dependency for ref_hallucination_arena (chart generation)
pip install matplotlib
```

## Gather from user before running

| Info | Required? | Notes |
|------|-----------|-------|
| Config YAML path | Yes | Defines endpoints, dataset, verification settings |
| Dataset path | Yes | JSON/JSONL file with queries (can be set in config) |
| API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. |
| CrossRef email | No | Improves API rate limits for verification |
| PubMed API key | No | Improves PubMed rate limits |
| Output directory | No | Default: `./evaluation_results/ref_hallucination_arena` |
| Report language | No | `"en"` (default) or `"zh"` |
| Tavily API key | No | Required only if using tool-augmented mode |

## Quick start

### CLI

```bash
# Run evaluation with config file
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Resume from checkpoint (default behavior)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Start fresh, ignore checkpoint
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save

# Override output directory
python -m cookbooks.ref_hallucination_arena --config config.yaml \
  --output_dir ./my_results --save
```

### Python API

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/academic-eval/03-ref-hallucination-arena/SKILL.md