# Evaluate Rag

Guides evaluation of RAG pipeline retrieval and generation quality.

- Repository: https://github.com/ai-evals-course/evals-skills (1700 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-24
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ai-evals-course/evals-skills --skill evaluate-rag -g -a claude-code -y`
Codex: `npx skills add ai-evals-course/evals-skills --skill evaluate-rag -g -a codex -y`
Cursor: `npx skills add ai-evals-course/evals-skills --skill evaluate-rag -g -a cursor -y`
Gemini CLI: `npx skills add ai-evals-course/evals-skills --skill evaluate-rag -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add ai-evals-course/evals-skills`

## Description

Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Evaluate RAG

## Overview

1. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.
2. Build a retrieval evaluation dataset: queries paired with relevant document chunks.
3. Measure retrieval quality with Recall@k (most important for first-pass retrieval).
4. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).
5. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.

## Prerequisites

Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.

## Core Instructions

### Evaluate Retrieval and Generation Separately

Measure each component independently. Use the appropriate metric for each retrieval stage:

- **First-pass retrieval:** Optimize for Recall@k. Include all relevant documents, even at the cost of noise.
- **Reranking:** Optimize for Precision@k, MRR, or NDCG@k. Rank the most relevant documents first.

### Building a Retrieval Evaluation Dataset

You need queries paired with ground-truth relevant document chunks.

**Manual curation (highest quality):** Write realistic questions and map each to the exact chunk(s) containing the answer.

**Synthetic QA generation (scalable):** For each document chunk, prompt an LLM to extract a fact and generate a question answerable only from that fact.

Synthetic QA prompt template:

```
Given a chunk of text, extract a specific, self-contained fact from it.
Then write a question that is directly and unambiguously answered
by that fact alone.

Return output in JSON format:
{ "fact": "...", "question": "..." }

Chunk: "{text_chunk}"
```

**Adversarial question generation:** Create harder queries that resemble content in multiple chunks but are only answered by one.

The rest: https://github.com/ai-evals-course/evals-skills/blob/HEAD/skills/evaluate-rag/SKILL.md