# Ab Test Setup

Use when designing an A/B or split test: define the hypothesis, control and variants...

- Repository: https://github.com/sickn33/agentic-awesome-skills (48984 stars)
- Category: Design
- License: MIT
- Last updated: 2026-11-01
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -g -a claude-code -y`
Codex: `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -g -a codex -y`
Cursor: `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -g -a cursor -y`
Gemini CLI: `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add sickn33/agentic-awesome-skills`

## Description

Use when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# A/B Test Setup

## 1️⃣ Purpose & Scope

Define an experiment that can answer a specific product question, and verify its assumptions before exposing users. This procedure cannot guarantee validity by itself.

- Documents the stopping rule
- Estimates sample needs under stated assumptions
- Makes the hypothesis and decision criteria reviewable

---

## 2️⃣ Pre-Requisites

You must have:

- A clear user problem
- Access to an analytics source
- Roughly estimated traffic volume

### Hypothesis Quality Checklist

A valid hypothesis includes:

- Observation or evidence
- Single, specific change
- Directional expectation
- Defined audience
- Measurable success criteria

---

## 3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

- Present the **final hypothesis**
- Specify:
  - Target audience
  - Primary metric
  - Expected direction of effect
  - Minimum Detectable Effect (MDE)

Use the hypothesis already agreed in the task. If a launch-critical choice is missing, present the concrete choice for confirmation while continuing independent analysis. Do not repeatedly request approval for a decision already authorized.

---

## 4️⃣ Assumptions & Validity Check (Mandatory)

Explicitly list assumptions about:

- Traffic stability
- User independence
- Metric reliability
- Randomization quality
- External factors (seasonality, campaigns, releases)

If assumptions are weak or violated:

- Warn the user
- Recommend delaying or redesigning the test

---

## 5️⃣ Test Type Selection

Choose the simplest valid test:

- **A/B Test**, single change, two variants
- **A/B/n Test**, multiple variants, higher traffic required
- **Multivariate Test (MVT)**, interaction effects, very high traffic
- **Split URL Test**, major structural changes

Default to **A/B** unless there is a clear reason otherwise.

---

## 6️⃣ Metrics Definition

#### Primary Metric (Mandatory)

The rest: https://github.com/sickn33/agentic-awesome-skills/blob/HEAD/plugins/agentic-awesome-skills-claude/skills/ab-test-setup/SKILL.md