Rag Eval
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to...
Install
npx skills add agentscope-ai/OpenJudge --skill rag-eval -g -a claude-code -y- Repository
- agentscope-ai/OpenJudge1k
- Category
- AI and agents
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,282 tokens
- Runs scripts
- Yes, review before use
What it does
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality, separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.
RAG Eval
Evaluate RAG systems by diagnosing retrieval and generation separately. A single "RAG accuracy" number hides whether the problem is finding the right documents or using them correctly. This skill separates them so you know what to fix.
When to Activate
- User has a RAG pipeline (retriever + generator) with traces
- User wants to know if their RAG system hallucinates
- User is optimizing chunking strategy and needs before/after comparison
- User wants to build a RAG evaluation dataset
Checklist
You MUST create a task for each item and complete them in order:
- Load RAG traces, validate query + context + answer triples
- Separate retrieval vs generation, determine which layers to evaluate
- Run faithfulness evaluation, is the answer grounded in retrieved docs?
- Run retrieval evaluation, are the right documents retrieved?
- Build diagnostic matrix, cross-tabulate to find root cause
- Output findings, prioritized issues with concrete fixes
Fast path: run the bundled script
Once each trace has a faithfulness judgment (and ideally a retrieval signal), build the
retrieval-vs-generation diagnostic matrix with the bundled, tested script
(scripts/rag_diagnostic.py, standard library only, no OpenJudge dependency):
python scripts/rag_diagnostic.py --traces traces.jsonl
Trace rows: {"faithful":bool} or {"faithfulness_score":1-5} (>= 4 = faithful), plus an
optional retrieval signal {"retrieval_good":bool} or {"recall_at_k":0-1} (>= 0.5 = good).
It prints the generation faithful/hallucinating split, the 2×2 matrix when a retrieval signal
is present, and the primary issue (retrieval vs generation). --self-test to verify it.
Steps below explain how to separate the layers and produce the faithfulness/retrieval signals (with OpenJudge graders or any judge).
Step 1: Load RAG Traces
The minimum data needed per trace:
