A/B Test Analyzer
Calculates significance and sample sizes for A/B results and next tests.
Install
npx skills add irinabuht12-oss/marketing-skills --skill ab-test-analyzer -g -a claude-code -y- Repository
- irinabuht12-oss/marketing-skills6.6k
- Category
- Marketing
- Updated
- 8d ago
- License
- See repository
- Context cost
- Lightabout 1,302 tokens
- Runs scripts
- No, instructions only
What it does
Statistical significance calculator for A/B test results with sample size requirements, segment breakdowns, and hypothesis generation. Use when feeding test results, checking statistical significance, calculating sample sizes, analyzing experiment outcomes, or generating next test ideas based on results. Platform: Google and Meta.
A/B Test Analyzer
Evaluate A/B test results with statistical rigor and generate actionable next steps.
Process
- Collect test data - Variations, sample sizes, conversions, time period
- Check validity - Runtime, sample size, peeking issues
- Calculate significance - Z-score, p-value, confidence interval
- Segment analysis - Device, source, new vs returning
- Interpret results - Statistical vs practical significance
- Generate hypotheses - "Why it worked" and next test ideas
Sample Size Calculator
n = 2 × (Zα/2 + Zβ)² × p(1-p) / δ²
Where:
- n = sample size per variation
- Zα/2 = 1.96 (95% confidence) or 2.58 (99%)
- Zβ = 0.84 (80% power) or 1.28 (90%)
- p = baseline conversion rate
- δ = minimum detectable effect (absolute)
Quick Reference (95% confidence, 80% power):
| Baseline CR | 10% Relative MDE | 20% Relative MDE |
|---|---|---|
| 2% | 78,000/var | 19,500/var |
| 5% | 30,000/var | 7,500/var |
| 10% | 14,300/var | 3,600/var |
Significance Calculation
Z-score = (pB - pA) / √(SE²_A + SE²_B)
SE = √(p × (1-p) / n)
If |Z| > 1.96 → Significant at 95%
If |Z| > 2.58 → Significant at 99%
Validity Checklist
- Ran for minimum 14 days (day-of-week effects)
- Met required sample size before concluding
- No "peeking" at results during test (inflates false positives to 40%+)
- Traffic split was random and consistent
- No external factors (holidays, outages, campaigns)
Bayesian vs Frequentist
| Use Case | Approach |
|---|---|
| Large samples (>10K), yes/no decision | Frequentist |
| Continuous monitoring, early stopping | Bayesian |
| Regulatory/legal requirements | Frequentist |
| Want probability statements | Bayesian |
Output Format
## A/B Test Analysis
### Test Summary
- **Test Name**: [Name]
- **Hypothesis**: [What you expected]
- **Runtime**: [X days]
- **Sample Size**: Control [n] / Variant [n]
