Eval Audit
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated...
Install
npx skills add ai-evals-course/evals-skills --skill eval-audit -g -a claude-code -y- Repository
- ai-evals-course/evals-skills1.7k
- Category
- AI and agents
- Updated
- 11d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 2,383 tokens
- Runs scripts
- No, instructions only
What it does
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
Eval Audit
Inspect an LLM eval pipeline and produce a prioritized list of problems with concrete next steps.
Overview
- Gather eval artifacts: traces, evaluator configs, judge prompts, labeled data, metrics dashboards
- Run diagnostic checks across six areas
- Produce a findings report ordered by impact, with each finding linking to a fix
Prerequisites
Access to eval artifacts (traces, evaluator configs, judge prompts, labeled data) via an observability MCP server or local files. If none exist, skip to "No Eval Infrastructure."
Connecting to Eval Infrastructure
Check whether the user has an observability MCP server connected (Phoenix, Braintrust, LangSmith, Truesight or similar). If available, use it to pull traces, evaluator definitions, and experiment results. If not, ask for local files: CSVs, JSON trace exports, notebooks, or evaluation scripts.
Diagnostic Checks
Work through each area below. Inspect available artifacts, determine whether the problem exists, and record a finding if it does.
Prioritize findings by impact on the user's product. Present the most impactful findings first.
1. Error Analysis
Check: Has the user done systematic error analysis on real or synthetic traces?
Look for: labeled trace datasets, failure category definitions, notes from trace review. If evaluators exist but no documented failure categories, error analysis was likely skipped.
Finding if missing: Evaluators built without error analysis measure generic qualities ("helpfulness", "coherence") instead of actual failure modes. Start with error-discovery, or generate-synthetic-data first if no traces exist.
See: Your AI Product Needs Evals, LLM Evals FAQ
Check: Were failure categories brainstormed or observed?
