# Eval Audit

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated...

- Repository: https://github.com/ai-evals-course/evals-skills (1700 stars)
- Category: AI and agents
- License: Apache-2.0
- Last updated: 2026-10-24
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ai-evals-course/evals-skills --skill eval-audit -g -a claude-code -y`
Codex: `npx skills add ai-evals-course/evals-skills --skill eval-audit -g -a codex -y`
Cursor: `npx skills add ai-evals-course/evals-skills --skill eval-audit -g -a cursor -y`
Gemini CLI: `npx skills add ai-evals-course/evals-skills --skill eval-audit -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add ai-evals-course/evals-skills`

## Description

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Eval Audit

Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.

## Overview

1. Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
2. Run diagnostic checks across six areas
3. Produce a findings report ordered by impact, with each finding linking to a fix

## Prerequisites

Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."

## Connecting to Eval Infrastructure

Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.

## Diagnostic Checks

Work through each area below. Inspect available artifacts, determine whether the problem exists, and record a finding if it does.

Prioritize findings by impact on the user's product. Present the most impactful findings first.

### 1. Error Analysis

**Check:** Has the user done systematic error analysis on real or synthetic traces?

Look for: labeled trace datasets, failure category definitions, notes from trace review. If evaluators exist but no documented failure categories, error analysis was likely skipped.

**Finding if missing:** Evaluators built without error analysis measure generic qualities ("helpfulness", "coherence") instead of actual failure modes. Start with `error-discovery`, or `generate-synthetic-data` first if no traces exist.

See: [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/index.html), [LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/)

**Check:** Were failure categories brainstormed or observed?

The rest: https://github.com/ai-evals-course/evals-skills/blob/HEAD/skills/eval-audit/SKILL.md