RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the openjudge library.

When to Use This Skill

• Building scalar rewards for GRPO / REINFORCE rollout scoring
• Generating (chosen, rejected) preference pairs for DPO / IPO
• Best-of-N candidate selection
• Multi-dimensional reward shaping (correctness + safety + format)
• Replacing or bootstrapping a reward model with LLM-as-judge

Step 1, Choose Your Reward Strategy

Use this decision tree before writing any code:

RL Algorithm + Task type?
│
├── GRPO / REINFORCE, Verifiable task (math, code, structured output)
│   └── → POINTWISE  ✅  (FunctionGrader, exact score, zero LLM cost)
│
├── GRPO / REINFORCE, Subjective task (instruction following, dialogue, summarization)
│   └── → PAIRWISE TOURNAMENT  ✅  (compare each rollout vs all others in group,
│                                    reward = net win rate within group)
│
├── DPO / IPO / SLiC, need (chosen, rejected) pairs
│   └── → PAIRWISE  ✅  (two-way comparison, return winner/loser)
│
└── Best-of-N / reranking, rank N candidates
    └── → LISTWISE  ✅  (single call ranks all N at once)

Cost constraint?
├── Low budget
│   └── FunctionGrader (free) → pointwise; or pairwise with small judge model
│
├── Medium budget
│   └── Pointwise: 2, 3 LLM graders + WeightedSumAggregator
│   └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group
│
└── High quality / no cost limit
    └── Pointwise voting (3, 5 calls) or pairwise with strong judge + debiasing

Sub-documents, Read When Relevant

Topic	File	Read when…
Pointwise multi-dim reward	pointwise.md	GRPO on verifiable tasks; multi-dimension scoring
Pairwise reward	pairwise.md	GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs