
Add Runner Evaluation
Adds a runner protocol evaluation with its roster, assertions and report fixture.
Install
npx skills add paperclipai/paperclip --skill add-runner-eval -g -a claude-code -y- Repository
- paperclipai/paperclip110k
- Category
- Testing
- Updated
- 3d ago
- License
- MIT
- Context cost
- Lightabout 990 tokens
- Runs scripts
- No, instructions only
What it does
Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
Add a Runner Eval
Use this skill for the Runner Evals family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in add-product-e2e-eval.
Locate the Paperclip checkout using PAPERCLIP_ROOT when supplied, or
git rev-parse --show-toplevel from a checkout. From outside Git, inspect the
workspace roots (for example ~/paperclipai/paperclip) and verify that the
selected root contains packages/paperclip-runner and tests/runner-e2e.
Locate paperclip-evals using PAPERCLIP_EVALS_ROOT or a discovered sibling;
a worktree's parent directory need not contain that repository. Read
doc/evals.md and packages/paperclip-runner/docs/runner-protocol-live-evals.md,
then inspect the nearest existing case, roster, schema, and report test before
editing. Definitions and authored cases belong in the sibling
paperclip-evals/evals/paperclip-runner; Runner integration, aggregation,
viewer, and publication behavior belongs in packages/paperclip-runner.
Keep the control-plane boundary explicit in names and documentation.
The sibling eval README is the concrete map: cases live under cases/,
company fixtures under fixtures/, runtime/model settings under configs/,
selections under rosters/, and maintained campaign membership under
campaigns/live-direct-full.json. Update inventory/coverage mappings when the
program requires them; a new file alone does not join the maintained campaign.
From the Evals repository root, adapt these provider-free checks to the case
and roster you changed. Run the reliability-plan validator only when that
separate plan changes:

