Analyze Exports Skill

Purpose

When a tau2-bench task returns reward 0.0 (or fails), the export file contains everything you need to understand why, but it's buried in tool-call summaries and conversation transcripts. This skill systematically analyzes the export to show:

1. Where it failed, the exact round (Assistant #X, User #Y) and tool-call sequence
2. What should have happened, ground truth from the task explanation
3. Why it failed, root cause (missing logic, wrong arguments, bad order, policy misunderstanding)
4. Was the docs available?, whether the KB had the answer the agent needed
5. Evidence, quotes and citations from both the transcript and source documents

Input Format

An export file (plain text) containing:

• RUN: and TASK: metadata
• ★ TASK EXPLANATION (success criteria, expected actions)
• 🛠 TOOLS LOADED (available tools and KB docs)
• ⇄ TOOL-CALL SUMMARY (OURS vs GROUND TRUTH comparison)
• Full conversation transcript with tool calls and results

Output Format & File Organization

Analysis reports are saved in a standardized folder structure for easy organization and review:

FAILURE_ANALYSIS/
├── analysis/
│   ├── hard_15/
│   │   ├── task_037_detailed_analysis.md
│   │   ├── task_038_detailed_analysis.md
│   │   ├── task_039_detailed_analysis.md
│   │   └── task_041_detailed_analysis.md
│   ├── hard_16/
│   │   ├── task_053_detailed_analysis.md
│   │   └── task_054_detailed_analysis.md
│   └── [other_categories]/
│       └── [task_id]_detailed_analysis.md
└── README.md (overview of all analyses)

File Naming Convention: [task_id]_detailed_analysis.md

Folder Organization: Group by difficulty level (hard_15, hard_16, etc.) or category to match your existing structure.

Each analysis file contains these sections:

Report Structure

You will receive a structured analysis report with these sections:

## Export Analysis: [TASK_ID]