<HARD-GATE> NO v0 grader deployed WITHOUT explicitly marking it as uncalibrated. NO synthetic labels, LLM can generate eval inputs, but labels MUST come from real system output + human judgment. NO principle without a source label documenting where it came from. </HARD-GATE>

Bootstrap

Cold-start an evaluation system when you have nothing. In 30 minutes you get a working v0 grader and a clear path to a calibrated, trustworthy evaluation.

Requires OpenJudge (pip install py-openjudge) for the grader generators (SimpleRubricsGenerator / IterativeRubricsGenerator). The interview, stratification, and calibration-roadmap methodology is SDK-independent.

Checklist

You MUST create a task for each item and complete them in order:

1. Understand the product, one-shot interview, not question-by-question
2. Generate v0 grader, use OpenJudge SimpleRubricsGenerator
3. Synthesize eval inputs, 30 inputs with 60/30/10 stratification
4. Run v0 evaluation, GradingRunner with the generated grader
5. Output roadmap, exactly how to reach 50 labels → calibrate

Step 1: Product Interview (One Shot)

Ask the user to describe their system in one go:

To bootstrap your evaluation, I need to understand what you're building.
Please describe (all at once):

- What does your system do? Who uses it?
- What are 3 examples of perfect outputs?
- What are 3 things the system must never do?
- What failures worry you most?

Don't drip-feed these questions. One prompt, one answer. If the user provides a spec doc or design document instead, read that directly.

Step 2: Generate v0 Grader

Use OpenJudge's SimpleRubricsGenerator to create a zero-shot grader from the product description: