Create Skill Test
Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable...
Install
npx skills add dotnet/skills --skill create-skill-test -g -a claude-code -y- Repository
- dotnet/skills5.8k
- Category
- Testing
- Updated
- 3d ago
- License
- MIT
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- No, instructions only
What it does
Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable gh-aw workflow packages in the dotnet/skills repository. Use when creating skill or workflow-package tests, writing evaluation stimuli, defining graders and rubrics, sizing an eval for statistical power, or setting up test fixture files. Handles the Vally eval.yaml schema, fixture organization, and overfitting avoidance. Do not use for running or debugging existing evals (use improve-skill-quality) nor for skills authoring (use create-skill).
Create Skill Test
Scaffold an evaluation spec (eval.yaml) for a skill, agent, or workflow package so it conforms to the Vally schema,
passes skill-validator check and check_eval_quality.py, is powerful enough to return a verdict,
and does not overfit to the skill's own wording.
When to Use
- Creating a new
eval.yamlfor a skill, agent, or workflow package - Adding stimuli to an existing eval
- Sizing an eval so the pass gate can actually be reached
- Setting up or repairing fixture files alongside an eval
- Reviewing whether rubric items and graders risk overfitting
When Not to Use
- Diagnosing a failing or regressed eval, use
improve-skill-quality - Modifying the skill-validator or the evaluation workflows
- Creating or editing
SKILL.mdfiles, usecreate-skill
Inputs
| Input | Required | Description |
|---|---|---|
| Skill or agent name | Yes | Must exist under plugins/<plugin>/skills/ or plugins/<plugin>/agents/ |
| Plugin name | Yes | e.g. dotnet-msbuild |
| Skill content | Yes | Read it, you cannot write non-overfitted rubric items without it |
| Scenario hypothesis | Yes | State the expected improvement for a preference case, or the invariant protected by a guard |
| Failure modes to discriminate | Recommended | Each becomes one distinct stimulus |
Workflow
Step 1: Prove the eval belongs to the target
Before writing YAML, state what the stimulus proves. A preference stimulus is necessary when the target should improve the answer, action, restraint, or validation result compared with the same model without the target. A non-voting activation contract or no-op guard is necessary when it protects a meaningful invariant, even if correct behavior is baseline-equivalent.
Do not add a stimulus when it measures:
