# Product Experiments

Help users design, execute, and analyze product experiments to validate hypotheses and...

- Repository: https://github.com/RefoundAI/lenny-skills (1456 stars)
- Category: Design
- License: MIT
- Last updated: 2026-07-16
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add RefoundAI/lenny-skills --skill product-experiments -g -a claude-code -y`
Codex: `npx skills add RefoundAI/lenny-skills --skill product-experiments -g -a codex -y`
Cursor: `npx skills add RefoundAI/lenny-skills --skill product-experiments -g -a cursor -y`
Gemini CLI: `npx skills add RefoundAI/lenny-skills --skill product-experiments -g -a gemini-cli -y`

## Description

Help users design, execute, and analyze product experiments to validate hypotheses and measure true incremental impact while avoiding common statistical pitfalls.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Product Experimentation Excellence

Drive measurable growth and mitigate risk through rigorous A/B testing and data-driven learning.

Help the user with product experimentation excellence using insights from 9 guests and posts across Lenny's Podcast and Newsletter.

## How to Help

1. **Hypothesis Definition** - Guide the user in drafting clear, falsifiable hypotheses based on user behavior theories.
2. **Experimental Design** - Help determine the right metrics, sample sizes, and guardrail metrics for a clean test.
3. **Statistical Analysis** - Support the interpretation of p-values, confidence intervals, and potential sample ratio mismatches.
4. **Strategic Evaluation** - Assist in deciding whether to ship, iterate, or kill a feature based on experiment results and long-term business impact.

## Core Principles

### Use long-term holdouts for true incrementality
Archie Abrams: "So we constantly will relook at an experiment a year later, see that the way the GMV curve for the distribution was different than we might've originally thought. And that'll actually change what we do from that previous experiment. And so there's a lot of longterm monitoring of experiments over these very long time horizons to both inform what those input metrics are and more importantly hold ourselves accountable to, did we actually move what we cared about, which is that longterm GMV, in the right way?"

Implement holdout groups for one or more years to distinguish between immediate growth and short-term pull-forward effects. This ensures you are measuring the genuine downstream business impact of changes.

The rest: https://github.com/RefoundAI/lenny-skills/blob/HEAD/skills/product-experiments/SKILL.md