<HARD-GATE> NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%. NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6. NO alignment conclusion WITHOUT all 5 bias checks completed. </HARD-GATE>

Align Human

Measure whether your automatic judge agrees with human judgment, detect where and why they disagree, and build a roadmap to reduce human review over time.

When to Activate

• You have a working judge/grader and 50+ human-labeled examples
• You want to know if the judge is trustworthy enough to replace human review
• You've noticed the judge's decisions being overturned by humans
• You're preparing to deploy an evaluation as a production gate

Checklist

You MUST create a task for each item and complete them in order:

1. Load paired data, match judge verdicts with human labels
2. Measure TPR/TNR, confusion matrix + per-stratum breakdown
3. Calculate agreement, Cohen's kappa, Gwet's AC1, systematic bias
4. Run bias detection, 5 systematic bias checks
5. Analyze disagreements, cluster patterns + diagnose root causes
6. Build human-reduction roadmap, 4-phase transition plan
7. Confirm and record, one confirmation, then write results

Fast path: run the bundled script

Don't hand-write the calibration statistics, that is exactly where subtle bugs hide. Run the bundled, tested script (scripts/calibration.py, standard library only, no OpenJudge dependency):

python scripts/calibration.py --pairs pairs.jsonl                 # one paired file, OR
python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty