Prompt Regression
Use when the user has changed a prompt (system prompt, RAG template, agent instruction...
Install
npx skills add agentscope-ai/OpenJudge --skill prompt-regression -g -a claude-code -y- Repository
- agentscope-ai/OpenJudge1k
- Category
- Testing
- Updated
- 5d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,688 tokens
- Runs scripts
- Yes, review before use
What it does
Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.
Prompt Regression
Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension.
When to Activate
- You changed the system prompt and want to verify it's actually better
- You're iterating on RAG answer templates
- You're optimizing agent step-by-step instructions
- You want data to support a prompt change decision
Checklist
You MUST create a task for each item and complete them in order:
- Load and analyze prompts, diff the baseline vs candidate
- Derive comparison dimensions, from the prompt changes + task type
- Select graders per dimension, pairwise, judge, or rule
- Run position-debiased comparison, swap-aggregate to eliminate order bias
- Compute statistics, win rates + bootstrap 95% CI per dimension
- Present results, per-dimension verdict with confidence intervals
Fast path: run the bundled script
Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get
wrong). Run the bundled, tested script (scripts/pairwise.py, standard library only, no
OpenJudge dependency):
python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline
Each comparison row: {"id","model_a","model_b","score","dimension"?} where score >= 0.5
means model_a won. Emit two rows per query with A/B swapped to debias position. The
script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict
(BETTER / WORSE / TIED / INSUFFICIENT_EVIDENCE / INCONCLUSIVE; exit 0 only if
better). --self-test to verify it.
