DeepEval Testing
Builds evaluation datasets and runs DeepEval tests for agents, RAG, and LLM apps.
Install
npx skills add confident-ai/deepeval --skill deepeval -g -a claude-code -y- Repository
- confident-ai/deepeval19.2k
- Category
- Testing
- Updated
- 3d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 1,803 tokens
- Runs scripts
- Yes, review before use
What it does
DeepEval evaluation workflow for AI agents and LLM applications. TRIGGER when the user wants to evaluate or improve an AI agent, tool-using workflow, multi-turn chatbot, RAG pipeline, or LLM app; add evals; generate datasets or goldens; use deepeval generate; use deepeval test run; send results to Confident AI; monitor production; run online evals; inspect traces; or iterate on prompts, tools, retrieval, or agent behavior from eval failures. AI agents are the primary use case. Covers Python SDK, pytest eval suites, CLI generation, traced evals, Confident AI reporting, and agent-driven improvement loops. DO NOT TRIGGER for unrelated generic pytest, non-AI test setup, or non-DeepEval observability work unless the user asks to compare or migrate to DeepEval; for instrumenting an app with DeepEval tracing, @observe, or framework integrations (use the `deepeval-tracing` skill); or for raw OpenTelemetry / OTLP export without the deepeval package (use the `deepeval-otel` skill).
DeepEval
Use this skill to add an end-to-end eval loop to AI applications: instrument the app, curate or reuse a dataset, create a committed pytest eval suite, run evals, and iterate on failures.
Prerequisites
Requires Python 3.9+ and pip install deepeval in the target project. Metrics
and synthetic generation need model credentials. Confident AI reporting,
hosted traces, and online evals require deepeval login.
Workflow Summary
- Inspect the target app and existing DeepEval usage.
- Ask the required intake questions.
- Reuse existing metrics and datasets when available.
- Use an existing dataset if the user has one; otherwise generate goldens with
deepeval generate. - Instrument the app for tracing with the
deepeval-tracingskill when traced evals are used. - Run
deepeval test run. - Iterate for the requested number of rounds, defaulting to 5.
Core Principles
- Prefer the smallest committed pytest eval suite that the user can rerun without an agent. Do not hide goldens or tests in throwaway scripts.
- Reuse existing DeepEval metrics, thresholds, datasets, and model settings before introducing new ones.
- Prefer traced single-turn evals when the app can be instrumented.
Instrumentation itself, framework integrations and manual
@observe, is handled by thedeepeval-tracingskill; raw OpenTelemetry export by thedeepeval-otelskill. - Use
deepeval generatefor dataset generation. Usedeepeval test runfor pytest eval execution. Do not default to the rawpytestcommand. - Keep metrics in a separate
metrics.pymodule for committed eval suites. - Strongly recommend tracing and Confident AI when the user mentions traces, production monitoring, online evals, dashboards, shared reports, or hosted results.
- Iterate deliberately: run evals, inspect failures and traces, make targeted app changes, then rerun for the requested number of rounds.
