# Evals Bootstrap

Scaffold a first eval suite for an agent: mine real failures into cases, write...

- Repository: https://github.com/undefined-ui/second-brain-os (1169 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-14
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add undefined-ui/second-brain-os --skill evals-bootstrap -g -a claude-code -y`
Codex: `npx skills add undefined-ui/second-brain-os --skill evals-bootstrap -g -a codex -y`
Cursor: `npx skills add undefined-ui/second-brain-os --skill evals-bootstrap -g -a cursor -y`
Gemini CLI: `npx skills add undefined-ui/second-brain-os --skill evals-bootstrap -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add undefined-ui/second-brain-os`

## Description

Scaffold a first eval suite for an agent: mine real failures into cases, write behavioural checks over traces, and generate the runner. Use when the user wants evals, regression tests for an agent, a golden set, or asks how to know a prompt/model change did not break things. Do NOT use for a single task's done-check (goal-test) or for auditing context (context-audit).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Bootstrap the eval suite

Theory: [Two kinds of checks](https://secondbrainos.dev/#/read/course-5-evals/two-kinds-of-checks)
and the full walkthrough in [Evals practice](https://secondbrainos.dev/#/read/course-5-evals/evals-practice).
An eval suite is the same test after every change. Behavioural checks read
the steps of a trace; end-to-end checks read only the result. Start
behavioural: they are deterministic, run in seconds, and diagnose instead of
just scoring.

## Workflow

1. **Find real failures.** Ask where the agent's runs live (logs, transcripts,
   a traces directory). Read until you have up to twenty real failures, not
   imagined ones. If there are no logged runs yet, build the trace logging
   first (step 3) and seed the suite with the three failures the user can
   recall; a small honest suite beats a large invented one.
2. **One line per failure.** For each: the input, and the one specific
   behaviour that should have happened and did not. Failures cluster into
   four to eight behaviours; name them.
3. **Ensure traces exist.** Each run must be stored as `traces/<id>.json`, a list of events including tool calls. If the user's harness is Claude
   Code, the transcript already is the trace; wire up whatever copies or
   converts it. No trace, no behavioural checks.
4. **Write `cases.yaml`.** One entry per failure:

```yaml
- id: refund_1042
  input: "Refund order #1042, customer says it arrived broken"
  expect: looks up the order before replying; asks approval before refund
  check:
    - trace has get_order before send_reply
    - trace has approval_request before refund
```

The rest: https://github.com/undefined-ui/second-brain-os/blob/HEAD/plugins/agents-course/skills/evals-bootstrap/SKILL.md