# Experimentation

Designs, runs, and reads A/B tests and growth experiments, hypothesis, sample size...

- Repository: https://github.com/cbrock84/headcount (2554 stars)
- Category: Design
- License: MIT
- Last updated: 2026-10-18
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add cbrock84/headcount --skill experimentation -g -a claude-code -y`
Codex: `npx skills add cbrock84/headcount --skill experimentation -g -a codex -y`
Cursor: `npx skills add cbrock84/headcount --skill experimentation -g -a cursor -y`
Gemini CLI: `npx skills add cbrock84/headcount --skill experimentation -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add cbrock84/headcount`

## Description

Designs, runs, and reads A/B tests and growth experiments, hypothesis, sample size, duration, and honest interpretation. Use this to plan a test, judge whether a result is real, build an experimentation program, decide what to test next, or diagnose why tests keep producing inconclusive or non-replicating results.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Experimentation

Most A/B testing programs produce confident conclusions from insufficient data. The discipline is
almost entirely in what you do before launch.

## Before running

- **Hypothesis with a mechanism.** "Moving the pricing table above the fold will raise trial starts,
  because visitors currently leave before seeing pricing." Not "let's try a green button."
- **One primary metric**, chosen in advance. Secondary metrics are context, never the verdict.
- **Sample size calculated in advance**, from your baseline rate and the smallest lift that would
  change a decision. If the required sample is unreachable, do not run the test, decide by judgment
  and say so.
- **Duration set in advance**, covering at least one full weekly cycle, and two if the buying cycle
  is long.
- **Guardrail metrics** that would make you reject a win: refunds, support volume, downstream
  retention.

## While running

Do not look at results and act on them mid-flight. Peeking and stopping at significance is the
single most common way to generate false positives, and it is very effective at it.

Check only that the test is running correctly, even split, no broken variant, tracking firing.

## Reading

- **At the pre-set duration**, not before, and not extended because it is nearly significant.
  Extending until significance manufactures it.
- **Significance is not size.** A statistically significant 0.3% lift may not be worth shipping.
- **Inconclusive is a real result** and the most common one. It means the change did not matter
  enough to detect, which is useful.
- **Check the guardrails** before declaring a win.
- **Segment afterward for hypotheses only**, never for verdicts. Slice enough ways and something is
  always significant.

## Program level

Test where the traffic and the leverage are. Most sites can only run a handful of adequately powered
tests a year, spend them on structural questions, not button colors.

The rest: https://github.com/cbrock84/headcount/blob/HEAD/plugins/demand-generation/skills/experimentation/SKILL.md