# 01 Graders And Pipeline

Build custom LLM evaluation pipelines using the OpenJudge framework.

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Design
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill 01-graders-and-pipeline -g -a gemini-cli -y`

## Description

Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# OpenJudge Skill

Build evaluation pipelines for LLM applications using the `openjudge` library.

## When to Use This Skill

- User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)
- User wants to compare two or more models and rank them
- User wants to design a scoring rubric and automate evaluation
- User wants to analyze evaluation results statistically
- User wants to build a reward model or quality filter

## Sub-documents, Read When Relevant

| Topic | File | Read when… |
|-------|------|------------|
| Grader selection & configuration | `graders.md` | User needs to pick or configure an evaluator |
| Batch evaluation pipeline | `pipeline.md` | User needs to run evaluation over a dataset |
| Auto-generate graders from data | `generator.md` | No rubric yet; generate from labeled examples |
| Analyze & compare results | `analyzer.md` | User wants win rates, statistics, or metrics |

Read the relevant sub-document **before** writing any code.

## Install

```bash
pip install py-openjudge
```

## Architecture Overview

```
Dataset (List[dict])
    │
    ▼
GradingRunner                    ← orchestrates everything
    │
    ├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
    ├─► Grader B ──► EvaluationStrategy ──► _aevaluate() ──► GraderScore / GraderRank
    └─► Grader C ...
    │
    ├─► Aggregator (optional)    ← combine multiple grader scores into one
    │
    └─► RunnerResult             ← {grader_name: [GraderScore, ...]}
            │
            ▼
        Analyzer                 ← statistics, win rates, validation metrics
```

## 5-Minute Quick Start

Evaluate responses for correctness using a built-in grader:

```python
import asyncio
from openjudge.models.openai_chat_model import OpenAIChatModel
from openjudge.graders.common.correctness import CorrectnessGrader
from openjudge.runner.grading_runner import GradingRunner

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/openjudge-core/01-graders-and-pipeline/SKILL.md