# Create Skill Test

Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable...

- Repository: https://github.com/dotnet/skills (5787 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add dotnet/skills --skill create-skill-test -g -a claude-code -y`
Codex: `npx skills add dotnet/skills --skill create-skill-test -g -a codex -y`
Cursor: `npx skills add dotnet/skills --skill create-skill-test -g -a cursor -y`
Gemini CLI: `npx skills add dotnet/skills --skill create-skill-test -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add dotnet/skills`

## Description

Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable gh-aw workflow packages in the dotnet/skills repository. Use when creating skill or workflow-package tests, writing evaluation stimuli, defining graders and rubrics, sizing an eval for statistical power, or setting up test fixture files. Handles the Vally eval.yaml schema, fixture organization, and overfitting avoidance. Do not use for running or debugging existing evals (use improve-skill-quality) nor for skills authoring (use create-skill).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Create Skill Test

Scaffold an evaluation spec (`eval.yaml`) for a skill, agent, or workflow package so it conforms to the Vally schema,
passes `skill-validator check` and `check_eval_quality.py`, is powerful enough to return a verdict,
and does not overfit to the skill's own wording.

## When to Use

- Creating a new `eval.yaml` for a skill, agent, or workflow package
- Adding stimuli to an existing eval
- Sizing an eval so the pass gate can actually be reached
- Setting up or repairing fixture files alongside an eval
- Reviewing whether rubric items and graders risk overfitting

## When Not to Use

- Diagnosing a failing or regressed eval, use `improve-skill-quality`
- Modifying the skill-validator or the evaluation workflows
- Creating or editing `SKILL.md` files, use `create-skill`

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| Skill or agent name | Yes | Must exist under `plugins/<plugin>/skills/` or `plugins/<plugin>/agents/` |
| Plugin name | Yes | e.g. `dotnet-msbuild` |
| Skill content | Yes | Read it, you cannot write non-overfitted rubric items without it |
| Scenario hypothesis | Yes | State the expected improvement for a preference case, or the invariant protected by a guard |
| Failure modes to discriminate | Recommended | Each becomes one distinct stimulus |

## Workflow

### Step 1: Prove the eval belongs to the target

Before writing YAML, state what the stimulus proves. A preference stimulus is necessary when the
target should improve the answer, action, restraint, or validation result compared with the same
model without the target. A non-voting activation contract or no-op guard is necessary when it
protects a meaningful invariant, even if correct behavior is baseline-equivalent.

Do not add a stimulus when it measures:

The rest: https://github.com/dotnet/skills/blob/HEAD/.agents/skills/create-skill-test/SKILL.md