# Metric Design

Use when the user has evaluation principles or a dataset but needs help choosing the...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Design
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill metric-design -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill metric-design -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill metric-design -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill metric-design -g -a gemini-cli -y`

## Description

Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Metric Design

Select, configure, and combine evaluation graders into a working pipeline. You choose
the right tool for each evaluation dimension, from zero-cost code checks to LLM judges, and produce executable `GradingRunner` code that runs on OpenJudge.

> **Requires OpenJudge** (`pip install py-openjudge`). This skill is intentionally
> SDK-centric, grader selection, `GradingRunner`, and aggregators are OpenJudge APIs. The
> design/decision logic still applies if you use another harness; only the code does not.

## When to Activate

- User has eval dimensions/principles but doesn't know which grader type to use
- User wants to write an LLM-as-judge prompt for a specific failure mode
- User needs a composite score combining multiple evaluation dimensions
- User wants to auto-generate graders from labeled data instead of writing them manually
- User's current evaluation is all LLM-based and too expensive/too slow

## Checklist

You MUST create a task for each item and complete them in order:

1. **Select grader types**, per dimension, pick the right grader class
2. **Create custom graders**, write judge prompts (4-component) or function graders
3. **Auto-generate if applicable**, use OpenJudge Generator for cold starts
4. **Run anti-pattern scan**, check for Likert, missing few-shot, vague criteria
5. **Build pipeline code**, assemble GradingRunner with graders + aggregators

## Step 1: Select Grader Type Per Dimension

For each evaluation dimension, walk this decision tree (first match wins):

```
1. Can a deterministic rule check this?
   → StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
   Examples: exact match for classification labels, regex for format checks,
             JSON schema validation, keyword presence/absence

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/02-metric-design/SKILL.md