# Skillgrade Setup

Sets up and runs skillgrade evaluation pipelines for AI agents and Agent Skills.

- Repository: https://github.com/mgechev/skillgrade (734 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-11
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add mgechev/skillgrade --skill skillgrade-setup -g -a claude-code -y`
Codex: `npx skills add mgechev/skillgrade --skill skillgrade-setup -g -a codex -y`
Cursor: `npx skills add mgechev/skillgrade --skill skillgrade-setup -g -a cursor -y`
Gemini CLI: `npx skills add mgechev/skillgrade --skill skillgrade-setup -g -a gemini-cli -y`

## Description

Sets up and runs skillgrade evaluation pipelines for AI agents and Agent Skills. Use when initializing eval configurations, running trials, reviewing results, exporting reports, or integrating with CI. Don't use for writing grader scripts, general test authoring, or non-agentic documentation.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Skillgrade Evaluation Setup

## Procedures

**Step 1: Install Skillgrade**
1. Verify Node.js 20+ and Docker are available.
2. Run `npm i -g skillgrade` to install the CLI globally.

**Step 2: Initialize an Eval Configuration**
1. Navigate to any project directory.
2. If the directory contains a `SKILL.md` and an API key is set, `skillgrade init` generates AI-powered tasks and graders tailored to the skill.
3. Without a `SKILL.md` or API key, `skillgrade init` creates a ready-to-edit generic eval template.
4. Pass `--force` to overwrite an existing `eval.yaml`.

**Step 3: Configure eval.yaml**
1. Read `references/eval-yaml-spec.md` for the full configuration schema.
2. Define tasks under the `tasks:` key. Each task requires:
   - `name`: unique task identifier
   - `instruction`: what the agent should accomplish (OR `turns:` for multi-turn evals)
   - `graders`: one or more scoring mechanisms (see the `skillgrade-graders` skill)
3. Optionally add `workspace:` (files the agent edits) and `context:` (reference material not directly graded).
4. Configure `defaults:` for agent, provider, trials, timeout, and threshold.
5. For multi-turn evals, replace `instruction` with `turns:`, each turn has its own `instruction` and optional `graders`. The workspace persists across turns.
6. For batch evals, add `dataset: path/to/file.jsonl` (or `.csv`) to a task. Each row expands into a separate task with `{{key}}` template variables interpolated in the instruction.

The rest: https://github.com/mgechev/skillgrade/blob/HEAD/skills/skillgrade-setup/SKILL.md