Evaluate Presets

Overview

Systematically test all hat collection presets using shell scripts. Direct CLI invocation, no meta-orchestration complexity.

When to Use

• Testing preset configurations after changes
• Auditing the preset library for quality
• Validating new presets work correctly
• After modifying hat routing logic

Quick Start

Evaluate a single preset:

./tools/evaluate-preset.sh tdd-red-green claude

Evaluate all presets:

./tools/evaluate-all-presets.sh claude

Arguments:

• First arg: preset name (without .yml extension)
• Second arg: backend (claude or kiro, defaults to claude)

Bash Tool Configuration

IMPORTANT: When invoking these scripts via the Bash tool, use these settings:

• Single preset evaluation: Use timeout: 600000 (10 minutes max) and run_in_background: true
• All presets evaluation: Use timeout: 600000 (10 minutes max) and run_in_background: true

Since preset evaluations can run for hours (especially the full suite), always run in background mode and use the TaskOutput tool to check progress periodically.

Example invocation pattern:

Bash tool with:
  command: "./tools/evaluate-preset.sh tdd-red-green claude"
  timeout: 600000
  run_in_background: true

After launching, use TaskOutput with block: false to check status without waiting for completion.

What the Scripts Do

evaluate-preset.sh

1. Loads test task from tools/preset-test-tasks.yml (if yq available)
2. Creates merged config with evaluation settings
3. Runs Ralph with --record-session for metrics capture
4. Captures output logs, exit codes, and timing
5. Extracts metrics: iterations, hats activated, events published