Error Discovery Skill

You are running an interactive error analysis session. The user has a dataset (JSONL, CSV, JSON, etc.) of LLM outputs or traces and wants to discover failure modes by reviewing samples.

This skill is meant for interactive sessions only.

This skill has two parts:

• This file covers phases 1 through 4. You read the data, design the UI, build it, and select samples.
• review-loop.md covers phase 5. You run the interactive review session.

Read this file first to build everything. Once the app is running and the human starts reviewing, follow review-loop.md.

Progress updates

Each phase takes time, especially building the interface. Tell the user what you are doing at each step. Before starting a phase, say what you are about to do and why. When a step finishes, say what you did and what comes next. For example: "Reading 10 sample records to understand the data shape", "Building the HTML app with three views: article, map, and progress", "Clustering on 5 features to select diverse samples." Do not go silent for long stretches.

Phase 1: Understand the domain and data

Before building anything, study the dataset thoroughly.

1a: Read and inventory the data

Load the file. Examine 5 to 10 records across the distribution. For each record, identify:

• All fields, their types, and what they represent
• Which fields are the primary content the human needs to judge
• Which fields are metadata (context for understanding, but not the thing being judged)
• Which fields vary across records and which are constant

1b: Identify the content structure

The data could take many forms. Determine which pattern fits: