# Eval Report

Use when the user has run multiple evaluation skills and wants a comprehensive analysis...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill eval-report -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill eval-report -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill eval-report -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill eval-report -g -a gemini-cli -y`

## Description

Use when the user has run multiple evaluation skills and wants a comprehensive analysis, maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

<HARD-GATE>
NO recommendation WITHOUT statistical evidence backing it.
NO "system ready" declaration WITHOUT all calibrated judges passing AND all production gates green.
NO trend analysis WITHOUT at least 2 data points in history.
</HARD-GATE>

# Eval Report

Synthesize everything from your evaluation journey into a comprehensive report.
This skill is read-only, it analyzes what exists, doesn't create new graders or datasets.

## When to Activate

- You've run 2+ evaluation skills and want the big picture
- You need to report evaluation status to non-technical stakeholders
- You're making a ship/no-ship decision and need evidence
- The evaluation system has been running for a while, time for a health check

## Checklist

You MUST create a task for each item and complete them in order:

1. **Inventory scan**, catalog everything in eval-design.md + runs/ history
2. **Maturity assessment**, 5 dimensions × 4 levels
3. **Cross-skill signal synthesis**, consistent findings + contradictions
4. **Weakness diagnosis**, failure concentration, correlations, stratum gaps
5. **Root cause classification**, system / metric / data / unclear
6. **Prioritized recommendations**, P0/P1/P2 actions with impact estimates
7. **Executive summary**, ship readiness + top 3 risks + next actions

## Step 1: Inventory Scan

Read `eval-design.md` and all `runs/` directories. Build a timeline:

```
Timeline:
  2026-04-15  01-eval-design   → 5 failure modes → 3 dimensions from 200 traces
  2026-04-18  02-metric-design → 4 graders configured (2 LLM + 1 rule + 1 executable)
  2026-04-25  (evaluation run) → 90-sample stratified dataset scored
  2026-05-01  03-align-human   → 2 judges Phase 3, 1 Phase 2, 1 Phase 1 (TPR/TNR + kappa)
  2026-05-10  07-redteam       → safety audit not yet run
```

Report key metrics:
- Total skills run, total principles, total labels
- Calibrated judges: X of Y (with TPR/TNR range)
- Last activity date per skill

## Step 2: Maturity Assessment

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/04-eval-report/SKILL.md