Generate Synthetic Data
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based...
Install
npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -g -a claude-code -y- Repository
- ai-evals-course/evals-skills1.7k
- Category
- Testing
- Updated
- 11d ago
- License
- Apache-2.0
- Context cost
- Lightabout 1,243 tokens
- Runs scripts
- No, instructions only
What it does
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.
Generate Synthetic Data
Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.
Prerequisites
Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.
Core Process
Step 1: Define Dimensions
Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.
Dimension 1: [Name], [What it captures]
Values: [value_a, value_b, value_c, ...]
Dimension 2: [Name], [What it captures]
Values: [value_a, value_b, value_c, ...]
Dimension 3: [Name], [What it captures]
Values: [value_a, value_b, value_c, ...]
Example for a real estate assistant:
Feature: what task the user wants
Values: [property search, scheduling, email drafting]
Client Persona: who the user serves
Values: [first-time buyer, investor, luxury buyer]
Scenario Type: query clarity
Values: [well-specified, ambiguous, out-of-scope]
Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.
Step 2: Draft 20 Tuples with the User
A tuple is one combination of dimension values defining a specific test case. Present 20 draft tuples to the user and iterate until they confirm the tuples reflect realistic scenarios. The user's domain knowledge is essential here, they know which combinations actually occur and which are unrealistic.
(Feature: Property Search, Persona: Investor, Scenario: Ambiguous)
(Feature: Scheduling, Persona: First-time Buyer, Scenario: Well-specified)
(Feature: Email Drafting, Persona: Luxury Buyer, Scenario: Out-of-scope)
Step 3: Generate More Tuples with an LLM
Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.
