Meta Eval
Use when the user wants to build an evaluation system for an LLM/agent application but...
Install
npx skills add agentscope-ai/OpenJudge --skill meta-eval -g -a claude-code -y- Repository
- agentscope-ai/OpenJudge1k
- Category
- Testing
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,381 tokens
- Runs scripts
- No, instructions only
What it does
Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start, they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.
Meta Eval
Entry router for the eval skill collection. You diagnose what the user has and route them to the right sub-skill. You don't do evaluation yourself, you're the triage desk.
Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data principles it needs, so it can be installed and used on its own.
Checklist
You MUST create a task for each item and complete them in order:
- Ask 4 diagnostic questions, data, labels, stakes, domain knowledge
- Match triage table, map user scenario to sub-skill
- Recommend sub-skill, tell the user which workflow to use and why
- Record routing decision, write a brief summary of what was diagnosed and recommended
Diagnostic Questions
Ask these 4 questions (all at once, don't drip-feed):
To route you to the right evaluation skill, I need to understand your situation:
1. What data do you have?
a) Agent traces / production logs
b) Product spec / design docs
c) Nothing yet, starting from scratch
2. Do you have human labels?
a) Yes, ≥50 labeled examples
b) Some, but fewer than 50
c) None
3. What are the stakes?
a) Low, internal experimentation, exploring options
b) Production, customer-facing, quality matters
c) Regulated, compliance requirements, audit trail needed
4. How well do you know this evaluation domain?
a) Very well, have clear standards and criteria
b) Somewhat, general idea but need structure
c) Not well, exploring what "good" even means


