# Error Discovery

Run error analysis on a dataset.

- Repository: https://github.com/ai-evals-course/evals-skills (1700 stars)
- Category: Design
- License: Apache-2.0
- Last updated: 2026-10-24
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ai-evals-course/evals-skills --skill error-discovery -g -a claude-code -y`
Codex: `npx skills add ai-evals-course/evals-skills --skill error-discovery -g -a codex -y`
Cursor: `npx skills add ai-evals-course/evals-skills --skill error-discovery -g -a cursor -y`
Gemini CLI: `npx skills add ai-evals-course/evals-skills --skill error-discovery -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add ai-evals-course/evals-skills`

## Description

Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Error Discovery Skill

You are running an interactive error analysis session. The user has a dataset (JSONL, CSV, JSON, etc.) of LLM outputs or traces and wants to discover failure modes by reviewing samples.

This skill is meant for interactive sessions only.

This skill has two parts:
- This file covers phases 1 through 4. You read the data, design the UI, build it, and select samples.
- [review-loop.md](review-loop.md) covers phase 5. You run the interactive review session.

Read this file first to build everything. Once the app is running and the human starts reviewing, follow [review-loop.md](review-loop.md).

## Progress updates

Each phase takes time, especially building the interface. Tell the user what you are doing at each step. Before starting a phase, say what you are about to do and why. When a step finishes, say what you did and what comes next. For example: "Reading 10 sample records to understand the data shape", "Building the HTML app with three views: article, map, and progress", "Clustering on 5 features to select diverse samples." Do not go silent for long stretches.

## Phase 1: Understand the domain and data

Before building anything, study the dataset thoroughly.

### 1a: Read and inventory the data

Load the file. Examine 5 to 10 records across the distribution. For each record, identify:
- All fields, their types, and what they represent
- Which fields are the primary content the human needs to judge
- Which fields are metadata (context for understanding, but not the thing being judged)
- Which fields vary across records and which are constant

### 1b: Identify the content structure

The data could take many forms. Determine which pattern fits:

The rest: https://github.com/ai-evals-course/evals-skills/blob/HEAD/skills/error-discovery/SKILL.md