Eval Report
Use when the user has run multiple evaluation skills and wants a comprehensive analysis...
Install
npx skills add agentscope-ai/OpenJudge --skill eval-report -g -a claude-code -y- Repository
- agentscope-ai/OpenJudge1k
- Category
- Engineering
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,329 tokens
- Runs scripts
- No, instructions only
What it does
Use when the user has run multiple evaluation skills and wants a comprehensive analysis, maturity assessment, cross-skill signals, trends, prioritized actions, and an executive summary. Also use when the user mentions eval health check, evaluation audit, ship readiness, evaluation maturity, or "how good is my evaluation system itself." This is a read-only analysis skill.
Eval Report
Synthesize everything from your evaluation journey into a comprehensive report. This skill is read-only, it analyzes what exists, doesn't create new graders or datasets.
When to Activate
- You've run 2+ evaluation skills and want the big picture
- You need to report evaluation status to non-technical stakeholders
- You're making a ship/no-ship decision and need evidence
- The evaluation system has been running for a while, time for a health check
Checklist
You MUST create a task for each item and complete them in order:
- Inventory scan, catalog everything in eval-design.md + runs/ history
- Maturity assessment, 5 dimensions × 4 levels
- Cross-skill signal synthesis, consistent findings + contradictions
- Weakness diagnosis, failure concentration, correlations, stratum gaps
- Root cause classification, system / metric / data / unclear
- Prioritized recommendations, P0/P1/P2 actions with impact estimates
- Executive summary, ship readiness + top 3 risks + next actions
Step 1: Inventory Scan
Read eval-design.md and all runs/ directories. Build a timeline:
Timeline:
2026-04-15 01-eval-design → 5 failure modes → 3 dimensions from 200 traces
2026-04-18 02-metric-design → 4 graders configured (2 LLM + 1 rule + 1 executable)
2026-04-25 (evaluation run) → 90-sample stratified dataset scored
2026-05-01 03-align-human → 2 judges Phase 3, 1 Phase 2, 1 Phase 1 (TPR/TNR + kappa)
2026-05-10 07-redteam → safety audit not yet run
Report key metrics:
- Total skills run, total principles, total labels
- Calibrated judges: X of Y (with TPR/TNR range)
- Last activity date per skill
