Metric Design

Select, configure, and combine evaluation graders into a working pipeline. You choose the right tool for each evaluation dimension, from zero-cost code checks to LLM judges, and produce executable GradingRunner code that runs on OpenJudge.

Requires OpenJudge (pip install py-openjudge). This skill is intentionally SDK-centric, grader selection, GradingRunner, and aggregators are OpenJudge APIs. The design/decision logic still applies if you use another harness; only the code does not.

When to Activate

• User has eval dimensions/principles but doesn't know which grader type to use
• User wants to write an LLM-as-judge prompt for a specific failure mode
• User needs a composite score combining multiple evaluation dimensions
• User wants to auto-generate graders from labeled data instead of writing them manually
• User's current evaluation is all LLM-based and too expensive/too slow

Checklist

You MUST create a task for each item and complete them in order:

1. Select grader types, per dimension, pick the right grader class
2. Create custom graders, write judge prompts (4-component) or function graders
3. Auto-generate if applicable, use OpenJudge Generator for cold starts
4. Run anti-pattern scan, check for Likert, missing few-shot, vague criteria
5. Build pipeline code, assemble GradingRunner with graders + aggregators

Step 1: Select Grader Type Per Dimension

For each evaluation dimension, walk this decision tree (first match wins):

1. Can a deterministic rule check this?
   → StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
   Examples: exact match for classification labels, regex for format checks,
             JSON schema validation, keyword presence/absence