
Evals Bootstrap
Scaffold a first eval suite for an agent: mine real failures into cases, write...
Install
npx skills add undefined-ui/second-brain-os --skill evals-bootstrap -g -a claude-code -y- Repository
- undefined-ui/second-brain-os1.2k
- Category
- Testing
- Updated
- 22d ago
- License
- MIT
- Context cost
- Lightabout 678 tokens
- Runs scripts
- No, instructions only
What it does
Scaffold a first eval suite for an agent: mine real failures into cases, write behavioural checks over traces, and generate the runner. Use when the user wants evals, regression tests for an agent, a golden set, or asks how to know a prompt/model change did not break things. Do NOT use for a single task's done-check (goal-test) or for auditing context (context-audit).
Bootstrap the eval suite
Theory: Two kinds of checks and the full walkthrough in Evals practice. An eval suite is the same test after every change. Behavioural checks read the steps of a trace; end-to-end checks read only the result. Start behavioural: they are deterministic, run in seconds, and diagnose instead of just scoring.
Workflow
- Find real failures. Ask where the agent's runs live (logs, transcripts, a traces directory). Read until you have up to twenty real failures, not imagined ones. If there are no logged runs yet, build the trace logging first (step 3) and seed the suite with the three failures the user can recall; a small honest suite beats a large invented one.
- One line per failure. For each: the input, and the one specific behaviour that should have happened and did not. Failures cluster into four to eight behaviours; name them.
- Ensure traces exist. Each run must be stored as
traces/<id>.json, a list of events including tool calls. If the user's harness is Claude Code, the transcript already is the trace; wire up whatever copies or converts it. No trace, no behavioural checks. - Write
cases.yaml. One entry per failure:
- id: refund_1042
input: "Refund order #1042, customer says it arrived broken"
expect: looks up the order before replying; asks approval before refund
check:
- trace has get_order before send_reply
- trace has approval_request before refund


