# Prompt Regression

Use when the user has changed a prompt (system prompt, RAG template, agent instruction...

- Repository: https://github.com/agentscope-ai/OpenJudge (1020 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-30
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add agentscope-ai/OpenJudge --skill prompt-regression -g -a claude-code -y`
Codex: `npx skills add agentscope-ai/OpenJudge --skill prompt-regression -g -a codex -y`
Cursor: `npx skills add agentscope-ai/OpenJudge --skill prompt-regression -g -a cursor -y`
Gemini CLI: `npx skills add agentscope-ai/OpenJudge --skill prompt-regression -g -a gemini-cli -y`

## Description

Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

<HARD-GATE>
NO conclusion about which prompt is better WITHOUT bootstrap 95% CI reported.
NO candidate declared "better" WITHOUT position-debiased (swap-aggregate) comparison.
NO comparison with fewer than 10 samples per axis, CI is too wide to be meaningful.
</HARD-GATE>

# Prompt Regression

Compare two prompts head-to-head and determine, with statistical rigor, whether
the candidate is better, worse, or tied on each evaluation dimension.

## When to Activate

- You changed the system prompt and want to verify it's actually better
- You're iterating on RAG answer templates
- You're optimizing agent step-by-step instructions
- You want data to support a prompt change decision

## Checklist

You MUST create a task for each item and complete them in order:

1. **Load and analyze prompts**, diff the baseline vs candidate
2. **Derive comparison dimensions**, from the prompt changes + task type
3. **Select graders per dimension**, pairwise, judge, or rule
4. **Run position-debiased comparison**, swap-aggregate to eliminate order bias
5. **Compute statistics**, win rates + bootstrap 95% CI per dimension
6. **Present results**, per-dimension verdict with confidence intervals

## Fast path: run the bundled script

Don't hand-write the win-rate + bootstrap math (the swap-aggregation and CI are easy to get
wrong). Run the bundled, tested script (`scripts/pairwise.py`, standard library only, **no
OpenJudge dependency**):

```bash
python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline
```

Each comparison row: `{"id","model_a","model_b","score","dimension"?}` where `score >= 0.5`
means `model_a` won. Emit two rows per query with A/B **swapped** to debias position. The
script reports per-dimension candidate/baseline/tie rates, bootstrap 95% CI, and a verdict
(`BETTER` / `WORSE` / `TIED` / `INSUFFICIENT_EVIDENCE` / `INCONCLUSIVE`; exit 0 only if
better). `--self-test` to verify it.

The rest: https://github.com/agentscope-ai/OpenJudge/blob/HEAD/skills/eval_pipeline/06-prompt-regression/SKILL.md