# AI Evals

Help users build robust infrastructure for measuring, monitoring, and iterating on AI...

- Repository: https://github.com/RefoundAI/lenny-skills (1456 stars)
- Category: AI and agents
- License: MIT
- Last updated: 2026-07-16
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add RefoundAI/lenny-skills --skill ai-evals -g -a claude-code -y`
Codex: `npx skills add RefoundAI/lenny-skills --skill ai-evals -g -a codex -y`
Cursor: `npx skills add RefoundAI/lenny-skills --skill ai-evals -g -a cursor -y`
Gemini CLI: `npx skills add RefoundAI/lenny-skills --skill ai-evals -g -a gemini-cli -y`

## Description

Help users build robust infrastructure for measuring, monitoring, and iterating on AI product performance using human, code-based, and LLM-as-a-judge methodologies.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# AI Evaluation Strategy

Move beyond vibe checks to systematic, empirical measurement of AI product quality and reliability.

Help the user with ai evaluation strategy using insights from 11 guests and posts across Lenny's Podcast and Newsletter.

## How to Help

1. **Identify Failure Modes** - Help the user conduct error analysis on real traces to find where the system specifically breaks.
2. **Select Eval Methods** - Recommend the right mix of human, code, and LLM judges based on the specific technical use case.
3. **Build Gold Sets** - Assist in curating a reference dataset of high-quality examples to act as the ground truth for your application.
4. **Operationalize** - Guide the user in integrating these evaluations into a CI/CD pipeline for continuous quality improvement.

## Core Principles

### Automate the Value Chain
Brendan Foody: "I think that for enterprises especially, the core way to think about it is how can they build a test or systematic way to measure how well AI automates their core value chain? So if it's an architecture firm that's producing these architecture diagrams of what they provide to their end customer, how can they effectively measure that? And each company has its own value chain or maybe a handful of them if it's a multi-product company."

Identify the core deliverables unique to your business and develop systematic tests to measure how accurately AI can replicate those specific tasks.

### Prioritize Subjective Excellence
Edwin Chen: "We are looking for a Nobel Prize-winning poetry. Is this poetry unique? Is it full of subtle imagery? Does it surprise you and target your heart? Does it teach you something about the nature of moonlight?"

True data quality is defined by deep, subjective human excellence, such as emotional resonance and uniqueness, rather than superficial binary checks.

The rest: https://github.com/RefoundAI/lenny-skills/blob/HEAD/skills/ai-evals/SKILL.md