# Bootstrap

Use when the user has nothing, no traces, no labels, no eval set, and needs to build...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill bootstrap -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill bootstrap -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill bootstrap -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill bootstrap -g -a gemini-cli -y`

## Description

Use when the user has nothing, no traces, no labels, no eval set, and needs to build a v0 evaluation from scratch. Also use when the user says "I need to start evaluating my app but don't know where to begin," "I want to set up eval for a new product," or has just identified failure modes and needs to turn them into principles. Outputs a v0 grader in 30 minutes using OpenJudge SimpleRubricsGenerator, plus a roadmap to reach calibrated evaluation.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

<HARD-GATE>
NO v0 grader deployed WITHOUT explicitly marking it as uncalibrated.
NO synthetic labels, LLM can generate eval inputs, but labels MUST come from real system output + human judgment.
NO principle without a source label documenting where it came from.
</HARD-GATE>

# Bootstrap

Cold-start an evaluation system when you have nothing. In 30 minutes you get a working
v0 grader and a clear path to a calibrated, trustworthy evaluation.

> **Requires OpenJudge** (`pip install py-openjudge`) for the grader generators
> (`SimpleRubricsGenerator` / `IterativeRubricsGenerator`). The interview, stratification,
> and calibration-roadmap methodology is SDK-independent.

## Checklist

You MUST create a task for each item and complete them in order:

1. **Understand the product**, one-shot interview, not question-by-question
2. **Generate v0 grader**, use OpenJudge SimpleRubricsGenerator
3. **Synthesize eval inputs**, 30 inputs with 60/30/10 stratification
4. **Run v0 evaluation**, GradingRunner with the generated grader
5. **Output roadmap**, exactly how to reach 50 labels → calibrate

## Step 1: Product Interview (One Shot)

Ask the user to describe their system in one go:

```
To bootstrap your evaluation, I need to understand what you're building.
Please describe (all at once):

- What does your system do? Who uses it?
- What are 3 examples of perfect outputs?
- What are 3 things the system must never do?
- What failures worry you most?
```

Don't drip-feed these questions. One prompt, one answer. If the user provides a spec
doc or design document instead, read that directly.

## Step 2: Generate v0 Grader

Use OpenJudge's `SimpleRubricsGenerator` to create a zero-shot grader from the
product description:

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/08-bootstrap/SKILL.md