Validate Evaluator

Calibrate an LLM judge against human judgment.

Overview

1. Split human-labeled data into train (10-20%), dev (40-45%), test (40-45%)
2. Run judge on dev set and measure TPR/TNR
3. Iterate on the judge until TPR and TNR > 90% on dev set
4. Run once on held-out test set for final TPR/TNR
5. Apply bias correction formula to production data

Prerequisites

• A built LLM judge prompt (from write-judge-prompt)
• Human-labeled data: ~100 traces with binary Pass/Fail labels per failure mode
  • Aim for ~50 Pass and ~50 Fail (balanced, even if real distribution is skewed)
  • Labels must come from a domain expert, not outsourced annotators
• Candidate few-shot examples from your labeled data

Core Instructions

Step 1: Create Data Splits

Split human-labeled data into three disjoint sets:

Split	Size	Purpose	Rules
Training	10-20% (~10-20 examples)	Source of few-shot examples for the judge prompt	Only clear-cut Pass and Fail cases. Used directly in the prompt.
Dev	40-45% (~40-45 examples)	Iterative evaluator refinement	Never include in the prompt. Evaluate against repeatedly.
Test	40-45% (~40-45 examples)	Final unbiased accuracy measurement	Do NOT look at during development. Used once at the end.

Target: 30-50 examples of each class (Pass and Fail) across dev and test combined. Use balanced splits even if real-world prevalence is skewed, you need enough Fail examples to measure TNR reliably.

from sklearn.model_selection import train_test_split

# First split: separate test set
train_dev, test = train_test_split(
    labeled_data, test_size=0.4, stratify=labeled_data['label'], random_state=42
)
# Second split: separate training examples from dev set
train, dev = train_test_split(
    train_dev, test_size=0.75, stratify=train_dev['label'], random_state=42
)
# Result: ~15% train, ~45% dev, ~40% test

Step 2: Run Evaluator on Dev Set