# Rag Eval

Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: AI and agents
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill rag-eval -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill rag-eval -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill rag-eval -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill rag-eval -g -a gemini-cli -y`

## Description

Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality, separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# RAG Eval

Evaluate RAG systems by diagnosing retrieval and generation separately. A single
"RAG accuracy" number hides whether the problem is finding the right documents or
using them correctly. This skill separates them so you know what to fix.

## When to Activate

- User has a RAG pipeline (retriever + generator) with traces
- User wants to know if their RAG system hallucinates
- User is optimizing chunking strategy and needs before/after comparison
- User wants to build a RAG evaluation dataset

## Checklist

You MUST create a task for each item and complete them in order:

1. **Load RAG traces**, validate query + context + answer triples
2. **Separate retrieval vs generation**, determine which layers to evaluate
3. **Run faithfulness evaluation**, is the answer grounded in retrieved docs?
4. **Run retrieval evaluation**, are the right documents retrieved?
5. **Build diagnostic matrix**, cross-tabulate to find root cause
6. **Output findings**, prioritized issues with concrete fixes

## Fast path: run the bundled script

Once each trace has a faithfulness judgment (and ideally a retrieval signal), build the
retrieval-vs-generation diagnostic matrix with the bundled, tested script
(`scripts/rag_diagnostic.py`, standard library only, **no OpenJudge dependency**):

```bash
python scripts/rag_diagnostic.py --traces traces.jsonl
```

Trace rows: `{"faithful":bool}` or `{"faithfulness_score":1-5}` (>= 4 = faithful), plus an
optional retrieval signal `{"retrieval_good":bool}` or `{"recall_at_k":0-1}` (>= 0.5 = good).
It prints the generation faithful/hallucinating split, the 2×2 matrix when a retrieval signal
is present, and the primary issue (retrieval vs generation). `--self-test` to verify it.

Steps below explain how to separate the layers and produce the faithfulness/retrieval signals
(with OpenJudge graders or any judge).

## Step 1: Load RAG Traces

The minimum data needed per trace:

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/05-rag-eval/SKILL.md