# Evaluate Presets

Use when testing Ralph's hat collection presets, validating preset configurations, or...

- Repository: https://github.com/mikeyobrien/ralph-orchestrator (3190 stars)
- Category: Design
- License: MIT
- Last updated: 2026-10-31
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add mikeyobrien/ralph-orchestrator --skill evaluate-presets -g -a claude-code -y`
Codex: `npx skills add mikeyobrien/ralph-orchestrator --skill evaluate-presets -g -a codex -y`
Cursor: `npx skills add mikeyobrien/ralph-orchestrator --skill evaluate-presets -g -a cursor -y`
Gemini CLI: `npx skills add mikeyobrien/ralph-orchestrator --skill evaluate-presets -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add mikeyobrien/ralph-orchestrator`

## Description

Use when testing Ralph's hat collection presets, validating preset configurations, or auditing the preset library for bugs and UX issues.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Evaluate Presets

## Overview

Systematically test all hat collection presets using shell scripts. Direct CLI invocation, no meta-orchestration complexity.

## When to Use

- Testing preset configurations after changes
- Auditing the preset library for quality
- Validating new presets work correctly
- After modifying hat routing logic

## Quick Start

**Evaluate a single preset:**
```bash
./tools/evaluate-preset.sh tdd-red-green claude
```

**Evaluate all presets:**
```bash
./tools/evaluate-all-presets.sh claude
```

**Arguments:**
- First arg: preset name (without `.yml` extension)
- Second arg: backend (`claude` or `kiro`, defaults to `claude`)

## Bash Tool Configuration

**IMPORTANT:** When invoking these scripts via the Bash tool, use these settings:

- **Single preset evaluation:** Use `timeout: 600000` (10 minutes max) and `run_in_background: true`
- **All presets evaluation:** Use `timeout: 600000` (10 minutes max) and `run_in_background: true`

Since preset evaluations can run for hours (especially the full suite), **always run in background mode** and use the `TaskOutput` tool to check progress periodically.

**Example invocation pattern:**
```
Bash tool with:
  command: "./tools/evaluate-preset.sh tdd-red-green claude"
  timeout: 600000
  run_in_background: true
```

After launching, use `TaskOutput` with `block: false` to check status without waiting for completion.

## What the Scripts Do

### `evaluate-preset.sh`

1. Loads test task from `tools/preset-test-tasks.yml` (if `yq` available)
2. Creates merged config with evaluation settings
3. Runs Ralph with `--record-session` for metrics capture
4. Captures output logs, exit codes, and timing
5. Extracts metrics: iterations, hats activated, events published

The rest: https://github.com/mikeyobrien/ralph-orchestrator/blob/HEAD/.claude/skills/evaluate-presets/SKILL.md