# Experiment Design

Framework for running single-variable cold email experiments.

- Repository: https://github.com/growthenginenowoslawski/coldoutboundskills (797 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-05
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add growthenginenowoslawski/coldoutboundskills --skill experiment-design -g -a claude-code -y`
Codex: `npx skills add growthenginenowoslawski/coldoutboundskills --skill experiment-design -g -a codex -y`
Cursor: `npx skills add growthenginenowoslawski/coldoutboundskills --skill experiment-design -g -a cursor -y`
Gemini CLI: `npx skills add growthenginenowoslawski/coldoutboundskills --skill experiment-design -g -a gemini-cli -y`

## Description

Framework for running single-variable cold email experiments. Defines experiment types (list-only, copy-only, combined), confidence weighting, minimum sample sizes, and success criteria. Use when the user wants to improve a campaign, test a new list vs old one, or compare copy variants. Prevents the #1 learning-loop mistake, changing list + copy + offer at once and losing all signal.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Experiment Design

If you change your list, your copy, and your offer at the same time, you learn nothing. This skill forces you to isolate one variable per experiment so you actually learn what's working.

## Why this exists

Most cold email operators run "throw-everything" experiments. Campaign 1 gets a new list, new copy, and a new offer. It works better. They declare victory. But they can't tell you WHY, was it the list? The copy? The offer?

Then campaign 2 changes all three again. Regression. Nobody knows why.

This skill is the antidote: plan each experiment around ONE variable, keep everything else constant, and confidence-weight the results.

## The three experiment types

### A. List-only experiment
- **What varies:** the list (targeting criteria)
- **What stays fixed:** copy, offer, sending infrastructure, sequence timing
- **What you learn:** whether this segment is a better fit than the baseline
- **Confidence on learnings:** HIGH on targeting, LOW on copy (because copy wasn't tested)

### B. Copy-only experiment
- **What varies:** the copy (subject, body, sequence, or A/B variant)
- **What stays fixed:** list, offer, infrastructure
- **What you learn:** whether this copy resonates with this audience
- **Confidence on learnings:** HIGH on copy, LOW on targeting

### C. Combined experiment (use sparingly)
- **What varies:** list AND copy (and sometimes offer)
- **What stays fixed:** only infrastructure
- **When to use:** launching a whole new campaign for a new ICP. You can't isolate because everything is new.
- **Confidence on learnings:** MEDIUM on everything. Use as hypothesis-generation, not conclusion.

## The Framework

### Step 1: Name your hypothesis

Every experiment starts with a one-sentence hypothesis:

> "Targeting Heads of Marketing at 50-200 person B2B SaaS companies will get a higher positive reply rate than our current VP Sales baseline, because [reason]."

Or:

The rest: https://github.com/growthenginenowoslawski/coldoutboundskills/blob/HEAD/skills/experiment-design/SKILL.md