# Eval Harness

Build a repeatable eval loop that grades agent output with an LLM judge, so...

- Repository: https://github.com/Archive228/loopkit (756 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-07-14
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Archive228/loopkit --skill eval-harness -g -a claude-code -y`
Codex: `npx skills add Archive228/loopkit --skill eval-harness -g -a codex -y`
Cursor: `npx skills add Archive228/loopkit --skill eval-harness -g -a cursor -y`
Gemini CLI: `npx skills add Archive228/loopkit --skill eval-harness -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add Archive228/loopkit`

## Description

Build a repeatable eval loop that grades agent output with an LLM judge, so prompt/skill changes get scored against a baseline instead of eyeballed. Reuses loopkit's verifier subagent as the grader, do not build a new one.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Eval Harness

Every prompt tweak in a long-running agent looks like an improvement in the moment. The only way to know is a graded run against fixed inputs. Loopkit already ships `.claude/agents/verifier.md`, that is your grader. Do not rebuild it.

## The three-stage loop

```
inputs.jsonl  →  runner  →  outputs.jsonl  →  verifier (per row)  →  verdicts.jsonl  →  diff vs baseline
```

Each stage writes to disk. No stage holds the whole run in context.

## Stage 1, inputs.jsonl

One JSON object per row: `{"id": "case-01", "input": "...", "expected": "..."}`.

- 20-100 cases is enough for a signal. More is nice, not required.
- Include known-hard cases, edge cases, and a couple of trivial ones as sanity anchors.
- Freeze the file. Rev the eval with a suffix (`inputs-v2.jsonl`) when you change it. Never edit in place, you lose the baseline.

## Stage 2, runner

A dumb loop: for each row, call the model with the current prompt/skill, capture output, write `{"id": ..., "output": ...}` to `outputs.jsonl`. No grading here, just capture.

- Same temperature every run (usually 0 for evals).
- Same seed / model version.
- Log the git SHA of the prompt/skill under test in the file header.

If the runner is smart it will bias the eval. Keep it dumb.

## Stage 3, verifier

Fan out one subagent per row (see `subagent-fanout`). Each gets:
- The input.
- The expected output (or spec).
- The actual output.
- The verifier system prompt from `.claude/agents/verifier.md`.

Verifier returns strict JSON: `{"pass": bool, "why": "..."}`. Collect into `verdicts.jsonl`.

## Diff vs baseline

Two runs of the same eval on two prompt versions → compare pass rates per case. What matters:

- **Overall pass rate**, the headline.
- **Regressions**, cases that were green and went red. These block ship.
- **New passes**, cases that were red and went green. These justify ship.
- **Flappy cases**, inconsistent across reruns. Investigate; may be genuine model nondeterminism or a bad case.

The rest: https://github.com/Archive228/loopkit/blob/HEAD/skills/eval-harness/SKILL.md