# Align Human

Use when the user has a judge/grader and human-labeled data, and wants to measure how...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill align-human -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill align-human -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill align-human -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill align-human -g -a gemini-cli -y`

## Description

Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

<HARD-GATE>
NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%.
NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6.
NO alignment conclusion WITHOUT all 5 bias checks completed.
</HARD-GATE>

# Align Human

Measure whether your automatic judge agrees with human judgment, detect where and
why they disagree, and build a roadmap to reduce human review over time.

## When to Activate

- You have a working judge/grader and 50+ human-labeled examples
- You want to know if the judge is trustworthy enough to replace human review
- You've noticed the judge's decisions being overturned by humans
- You're preparing to deploy an evaluation as a production gate

## Checklist

You MUST create a task for each item and complete them in order:

1. **Load paired data**, match judge verdicts with human labels
2. **Measure TPR/TNR**, confusion matrix + per-stratum breakdown
3. **Calculate agreement**, Cohen's kappa, Gwet's AC1, systematic bias
4. **Run bias detection**, 5 systematic bias checks
5. **Analyze disagreements**, cluster patterns + diagnose root causes
6. **Build human-reduction roadmap**, 4-phase transition plan
7. **Confirm and record**, one confirmation, then write results

## Fast path: run the bundled script

Don't hand-write the calibration statistics, that is exactly where subtle bugs hide. Run
the bundled, tested script (`scripts/calibration.py`, standard library only, **no OpenJudge
dependency**):

```bash
python scripts/calibration.py --pairs pairs.jsonl                 # one paired file, OR
python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty
```

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/03-align-human/SKILL.md