# 02 Rl Reward

Build RL reward signals using the OpenJudge framework.

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill 02-rl-reward -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill 02-rl-reward -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill 02-rl-reward -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill 02-rl-reward -g -a gemini-cli -y`

## Description

Build RL reward signals using the OpenJudge framework. Covers choosing between pointwise and pairwise reward strategies based on RL algorithm, task type, and cost; aggregating multi-dimensional pointwise scores into a scalar reward; pairwise tournament reward for GRPO on subjective tasks (net win rate across group rollouts); generating preference pairs for DPO/RLAIF; and normalizing scores for training stability. Use when building reward models, scoring rollouts for GRPO/REINFORCE, generating preference data for DPO, or doing Best-of-N selection.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and
reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.

## When to Use This Skill

- Building scalar rewards for GRPO / REINFORCE rollout scoring
- Generating (chosen, rejected) preference pairs for DPO / IPO
- Best-of-N candidate selection
- Multi-dimensional reward shaping (correctness + safety + format)
- Replacing or bootstrapping a reward model with LLM-as-judge

## Step 1, Choose Your Reward Strategy

Use this decision tree **before** writing any code:

```
RL Algorithm + Task type?
│
├── GRPO / REINFORCE, Verifiable task (math, code, structured output)
│   └── → POINTWISE  ✅  (FunctionGrader, exact score, zero LLM cost)
│
├── GRPO / REINFORCE, Subjective task (instruction following, dialogue, summarization)
│   └── → PAIRWISE TOURNAMENT  ✅  (compare each rollout vs all others in group,
│                                    reward = net win rate within group)
│
├── DPO / IPO / SLiC, need (chosen, rejected) pairs
│   └── → PAIRWISE  ✅  (two-way comparison, return winner/loser)
│
└── Best-of-N / reranking, rank N candidates
    └── → LISTWISE  ✅  (single call ranks all N at once)
```

```
Cost constraint?
├── Low budget
│   └── FunctionGrader (free) → pointwise; or pairwise with small judge model
│
├── Medium budget
│   └── Pointwise: 2, 3 LLM graders + WeightedSumAggregator
│   └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group
│
└── High quality / no cost limit
    └── Pointwise voting (3, 5 calls) or pairwise with strong judge + debiasing
```

## Sub-documents, Read When Relevant

| Topic | File | Read when… |
|-------|------|------------|
| Pointwise multi-dim reward | `pointwise.md` | GRPO on verifiable tasks; multi-dimension scoring |
| Pairwise reward | `pairwise.md` | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/openjudge-core/02-rl-reward/SKILL.md