Create Custom Grader

Convert team-owned benchmark definitions into runnable SkillEvaluator custom graders and, when needed, native Harbor tasks.

Purpose

Help an agent author valid SkillEvaluator BYOG/BYOT files from a user's benchmark instead of leaving the user with empty grader templates.

When To Use

Use this skill when the user wants to:

• bring an existing benchmark into SkillEvaluator
• turn a rubric into evals/grader.py or evals/grader.sh
• add custom metrics beside the default evaluator metrics
• convert task files such as task.yaml, task.json, pytest checks, or shell verifiers into BYOG or BYOT
• prove a team can run its own benchmark through SkillEvaluator

Do not use this skill for ordinary evals/evals.json authoring when no custom grading logic is needed. Use the normal dataset authoring workflow for that.

Instructions

1. Read the target skill, existing evals/, benchmark prompts, fixtures, and any verifier code.
2. Choose default_plus_custom when custom metrics should complement default evaluator scoring.
3. Choose custom_only only when the user wants the custom grader to own pass/fail semantics.
4. Write or update evals/grader.py or evals/grader.sh, then validate the Harbor contract.

Examples

skillevaluator init-custom-grader <skill-dir> --language python --mode default_plus_custom
skillevaluator tier3 validate <skill-dir>

Prerequisites

• The target skill directory should contain SKILL.md.
• The SkillEvaluator CLI should be available as skillevaluator.
• Full E2E evaluation may need agent credentials, sandbox access, GPU access, or service credentials depending on the benchmark.

Core Choice

Choose one path before writing files: