Skills

Meta Eval

Use when the user wants to build an evaluation system for an LLM/agent application but...

Install

npx skills add agentscope-ai/OpenJudge --skill meta-eval -g -a claude-code -y
Category
Testing
Updated
5d ago
License
Apache-2.0
Context cost
Mediumabout 2,381 tokens
Runs scripts
No, instructions only

What it does

Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start, they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.

<HARD-GATE> NO sub-skill recommendation WITHOUT identifying data_form + label_status (these two pick the entry workflow). ALWAYS give a provisional recommendation once data_form + label_status are known, even if stakes/user_prior are still unknown, then ask the remaining questions to refine the downstream path. Do not withhold the route while waiting on stakes. </HARD-GATE>

Meta Eval

Entry router for the eval skill collection. You diagnose what the user has and route them to the right sub-skill. You don't do evaluation yourself, you're the triage desk.

Each sub-skill is self-contained: it carries inline the data shapes, statistics, and data principles it needs, so it can be installed and used on its own.

Checklist

You MUST create a task for each item and complete them in order:

  1. Ask 4 diagnostic questions, data, labels, stakes, domain knowledge
  2. Match triage table, map user scenario to sub-skill
  3. Recommend sub-skill, tell the user which workflow to use and why
  4. Record routing decision, write a brief summary of what was diagnosed and recommended

Diagnostic Questions

Ask these 4 questions (all at once, don't drip-feed):

To route you to the right evaluation skill, I need to understand your situation:

1. What data do you have?
   a) Agent traces / production logs
   b) Product spec / design docs
   c) Nothing yet, starting from scratch

2. Do you have human labels?
   a) Yes, ≥50 labeled examples
   b) Some, but fewer than 50
   c) None

3. What are the stakes?
   a) Low, internal experimentation, exploring options
   b) Production, customer-facing, quality matters
   c) Regulated, compliance requirements, audit trail needed

4. How well do you know this evaluation domain?
   a) Very well, have clear standards and criteria
   b) Somewhat, general idea but need structure
   c) Not well, exploring what "good" even means
Read the whole SKILL.md on GitHub

More in Testing