# Bat Story Eval

Compare MCP tool behavior between target and baseline versions using pre-built and...

- Repository: https://github.com/homeassistant-ai/ha-mcp (5395 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add homeassistant-ai/ha-mcp --skill bat-story-eval -g -a claude-code -y`
Codex: `npx skills add homeassistant-ai/ha-mcp --skill bat-story-eval -g -a codex -y`
Cursor: `npx skills add homeassistant-ai/ha-mcp --skill bat-story-eval -g -a cursor -y`
Gemini CLI: `npx skills add homeassistant-ai/ha-mcp --skill bat-story-eval -g -a gemini-cli -y`

## Description

Compare MCP tool behavior between target and baseline versions using pre-built and custom stories with diff-based triage.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# BAT Story Evaluation

You are the evaluator. Run the steps in order; each one uses the output of the step before it.

## Parse Arguments

From `$ARGUMENTS`, extract:
- `--baseline`: REQUIRED. Git tag/branch of the released version (e.g., `v6.6.1`).
- `--agents`: Agent list (default: `gemini`). Comma-separated.
- `--stories`: Force specific pre-built story IDs (e.g., `s01,s02`). Overrides triage selection.
- `--all-stories`: Skip triage, run ALL pre-built stories.
- `--keep-container`: Keep HA containers alive after run for manual inspection.
- `--model`: Model for Claude agent (e.g., `haiku`, `sonnet`).

If `$ARGUMENTS` is `--help` or missing `--baseline`, show usage and stop:
```
/bat-story-eval --baseline v6.6.1
/bat-story-eval --baseline v6.6.1 --agents gemini,claude
/bat-story-eval --baseline v6.6.1 --stories s01,s02
/bat-story-eval --baseline v6.6.1 --all-stories --agents claude --model haiku
```

## Step 0: Triage (Diff Analysis + Custom Story Design)

### 0a. Compute Diff

```bash
cd "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")/worktree/uat-stories"
git diff <baseline>..HEAD -- src/ha_mcp/ --stat
git diff <baseline>..HEAD -- src/ha_mcp/ --name-only
```

Classify changed files:
- **Tool modules** (`tools/tools_*.py`): specific tool implementations changed
- **Core code** (`client/`, `server.py`, `errors.py`, `tools/util_helpers.py`): affects all tools
- **Utilities** (`utils/`, `resources/`): may affect all tools
- **No src/ changes**: only tests/docs/config, select 2 smoke-test stories

### 0b. Select Pre-built Stories

Skip if `--stories` or `--all-stories` was passed.

The rest: https://github.com/homeassistant-ai/ha-mcp/blob/HEAD/.claude/skills/bat-story-eval/SKILL.md