# Create Custom Grader

Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain...

- Repository: https://github.com/NVIDIA/SkillEvaluator (620 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -g -a claude-code -y`
Codex: `npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -g -a codex -y`
Cursor: `npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -g -a cursor -y`
Gemini CLI: `npx skills add NVIDIA/SkillEvaluator --skill create-custom-grader -g -a gemini-cli -y`

## Description

Use when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Create Custom Grader

Convert team-owned benchmark definitions into runnable SkillEvaluator custom graders
and, when needed, native Harbor tasks.

## Purpose

Help an agent author valid SkillEvaluator BYOG/BYOT files from a user's
benchmark instead of leaving the user with empty grader templates.

## When To Use

Use this skill when the user wants to:

- bring an existing benchmark into SkillEvaluator
- turn a rubric into `evals/grader.py` or `evals/grader.sh`
- add custom metrics beside the default evaluator metrics
- convert task files such as `task.yaml`, `task.json`, pytest checks, or shell
  verifiers into BYOG or BYOT
- prove a team can run its own benchmark through SkillEvaluator

Do not use this skill for ordinary `evals/evals.json` authoring when no custom
grading logic is needed. Use the normal dataset authoring workflow for that.

## Instructions

1. Read the target skill, existing `evals/`, benchmark prompts, fixtures, and any verifier code.
2. Choose `default_plus_custom` when custom metrics should complement default evaluator scoring.
3. Choose `custom_only` only when the user wants the custom grader to own pass/fail semantics.
4. Write or update `evals/grader.py` or `evals/grader.sh`, then validate the Harbor contract.

## Examples

```bash
skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
skillevaluator tier3 validate <skill-dir>
```

## Prerequisites

- The target skill directory should contain `SKILL.md`.
- The SkillEvaluator CLI should be available as `skillevaluator`.
- Full E2E evaluation may need agent credentials, sandbox access, GPU access, or
  service credentials depending on the benchmark.

## Core Choice

Choose one path before writing files:

The rest: https://github.com/NVIDIA/SkillEvaluator/blob/HEAD/src/skillevaluator/tier3/reference_skills/create-custom-grader/SKILL.md