Skillgrade Evaluation Setup

Procedures

Step 1: Install Skillgrade

1. Verify Node.js 20+ and Docker are available.
2. Run npm i -g skillgrade to install the CLI globally.

Step 2: Initialize an Eval Configuration

1. Navigate to any project directory.
2. If the directory contains a SKILL.md and an API key is set, skillgrade init generates AI-powered tasks and graders tailored to the skill.
3. Without a SKILL.md or API key, skillgrade init creates a ready-to-edit generic eval template.
4. Pass --force to overwrite an existing eval.yaml.

Step 3: Configure eval.yaml

1. Read references/eval-yaml-spec.md for the full configuration schema.
2. Define tasks under the tasks: key. Each task requires:
   • name: unique task identifier
   • instruction: what the agent should accomplish (OR turns: for multi-turn evals)
   • graders: one or more scoring mechanisms (see the skillgrade-graders skill)
3. Optionally add workspace: (files the agent edits) and context: (reference material not directly graded).
4. Configure defaults: for agent, provider, trials, timeout, and threshold.
5. For multi-turn evals, replace instruction with turns:, each turn has its own instruction and optional graders. The workspace persists across turns.
6. For batch evals, add dataset: path/to/file.jsonl (or .csv) to a task. Each row expands into a separate task with {{key}} template variables interpolated in the instruction.