# Eval Harness

Formal evaluation framework for Claude Code sessions implementing eval-driven...

- Repository: https://github.com/sangrokjung/claude-forge (868 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-09-03
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add sangrokjung/claude-forge --skill eval-harness -g -a claude-code -y`
Codex: `npx skills add sangrokjung/claude-forge --skill eval-harness -g -a codex -y`
Cursor: `npx skills add sangrokjung/claude-forge --skill eval-harness -g -a cursor -y`
Gemini CLI: `npx skills add sangrokjung/claude-forge --skill eval-harness -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add sangrokjung/claude-forge`

## Description

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Eval Harness Skill

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

## Philosophy

Eval-Driven Development treats evals as the "unit tests of AI development":
- Define expected behavior BEFORE implementation
- Run evals continuously during development
- Track regressions with each change
- Use pass@k metrics for reliability measurement

## Eval Types

### Capability Evals
Test if Claude can do something it couldn't before:
```markdown
[CAPABILITY EVAL: feature-name]
Task: Description of what Claude should accomplish
Success Criteria:
  - [ ] Criterion 1
  - [ ] Criterion 2
  - [ ] Criterion 3
Expected Output: Description of expected result
```

### Regression Evals
Ensure changes don't break existing functionality:
```markdown
[REGRESSION EVAL: feature-name]
Baseline: SHA or checkpoint name
Tests:
  - existing-test-1: PASS/FAIL
  - existing-test-2: PASS/FAIL
  - existing-test-3: PASS/FAIL
Result: X/Y passed (previously Y/Y)
```

## Grader Types

### 1. Code-Based Grader
Deterministic checks using code:
```bash
# Check if file contains expected pattern
grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"

# Check if tests pass
npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"

# Check if build succeeds
npm run build && echo "PASS" || echo "FAIL"
```

### 2. Model-Based Grader
Use Claude to evaluate open-ended outputs:
```markdown
[MODEL GRADER PROMPT]
Evaluate the following code change:
1. Does it solve the stated problem?
2. Is it well-structured?
3. Are edge cases handled?
4. Is error handling appropriate?

Score: 1-5 (1=poor, 5=excellent)
Reasoning: [explanation]
```

### 3. Human Grader
Flag for manual review:
```markdown
[HUMAN REVIEW REQUIRED]
Change: Description of what changed
Reason: Why human review is needed
Risk Level: LOW/MEDIUM/HIGH
```

## Metrics

The rest: https://github.com/sangrokjung/claude-forge/blob/HEAD/skills/eval-harness/SKILL.md