# Generate Synthetic Data

Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based...

- Repository: https://github.com/ai-evals-course/evals-skills (1700 stars)
- Category: Testing
- License: Apache-2.0
- Last updated: 2026-10-24
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -g -a claude-code -y`
Codex: `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -g -a codex -y`
Cursor: `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -g -a cursor -y`
Gemini CLI: `npx skills add ai-evals-course/evals-skills --skill generate-synthetic-data -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add ai-evals-course/evals-skills`

## Description

Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Generate Synthetic Data

Generate diverse, realistic test inputs that cover the failure space of an LLM pipeline.

## Prerequisites

Before generating synthetic data, identify where the pipeline is likely to fail. Ask the user about known failure-prone areas, review existing user feedback, or form hypotheses from available traces. Dimensions (Step 1) must target anticipated failures, not arbitrary variation.

## Core Process

### Step 1: Define Dimensions

Dimensions are axes of variation specific to your application. Choose dimensions based on where you expect failures.

```
Dimension 1: [Name], [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 2: [Name], [What it captures]
  Values: [value_a, value_b, value_c, ...]

Dimension 3: [Name], [What it captures]
  Values: [value_a, value_b, value_c, ...]
```

Example for a real estate assistant:

```
Feature: what task the user wants
  Values: [property search, scheduling, email drafting]

Client Persona: who the user serves
  Values: [first-time buyer, investor, luxury buyer]

Scenario Type: query clarity
  Values: [well-specified, ambiguous, out-of-scope]
```

Start with 3 dimensions. Add more only if initial traces reveal failure patterns along new axes.

### Step 2: Draft 20 Tuples with the User

A tuple is one combination of dimension values defining a specific test case. Present 20 draft tuples to the user and iterate until they confirm the tuples reflect realistic scenarios. The user's domain knowledge is essential here, they know which combinations actually occur and which are unrealistic.

```
(Feature: Property Search, Persona: Investor, Scenario: Ambiguous)
(Feature: Scheduling, Persona: First-time Buyer, Scenario: Well-specified)
(Feature: Email Drafting, Persona: Luxury Buyer, Scenario: Out-of-scope)
```

### Step 3: Generate More Tuples with an LLM

```
Generate 10 random combinations of ({dim1}, {dim2}, {dim3})
for a {your application description}.

The rest: https://github.com/ai-evals-course/evals-skills/blob/HEAD/skills/generate-synthetic-data/SKILL.md