Skills

Add Runner Evaluation

Adds a runner protocol evaluation with its roster, assertions and report fixture.

Install

npx skills add paperclipai/paperclip --skill add-runner-eval -g -a claude-code -y
Repository
paperclipai/paperclip110k
Category
Testing
Updated
3d ago
License
MIT
Context cost
Lightabout 990 tokens
Runs scripts
No, instructions only

What it does

Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.

Add a Runner Eval

Use this skill for the Runner Evals family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in add-product-e2e-eval.

Locate the Paperclip checkout using PAPERCLIP_ROOT when supplied, or git rev-parse --show-toplevel from a checkout. From outside Git, inspect the workspace roots (for example ~/paperclipai/paperclip) and verify that the selected root contains packages/paperclip-runner and tests/runner-e2e. Locate paperclip-evals using PAPERCLIP_EVALS_ROOT or a discovered sibling; a worktree's parent directory need not contain that repository. Read doc/evals.md and packages/paperclip-runner/docs/runner-protocol-live-evals.md, then inspect the nearest existing case, roster, schema, and report test before editing. Definitions and authored cases belong in the sibling paperclip-evals/evals/paperclip-runner; Runner integration, aggregation, viewer, and publication behavior belongs in packages/paperclip-runner. Keep the control-plane boundary explicit in names and documentation.

The sibling eval README is the concrete map: cases live under cases/, company fixtures under fixtures/, runtime/model settings under configs/, selections under rosters/, and maintained campaign membership under campaigns/live-direct-full.json. Update inventory/coverage mappings when the program requires them; a new file alone does not join the maintained campaign. From the Evals repository root, adapt these provider-free checks to the case and roster you changed. Run the reliability-plan validator only when that separate plan changes:

Read the whole SKILL.md on GitHub

More in Testing