# Experimental Design

Best practices for designing reproducible ML experiments.

- Repository: https://github.com/aiming-lab/AutoResearchClaw (14798 stars)
- Category: Design
- License: MIT
- Last updated: 2026-08-19
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add aiming-lab/AutoResearchClaw --skill experimental-design -g -a claude-code -y`
Codex: `npx skills add aiming-lab/AutoResearchClaw --skill experimental-design -g -a codex -y`
Cursor: `npx skills add aiming-lab/AutoResearchClaw --skill experimental-design -g -a cursor -y`
Gemini CLI: `npx skills add aiming-lab/AutoResearchClaw --skill experimental-design -g -a gemini-cli -y`

## Description

Best practices for designing reproducible ML experiments. Use when planning ablations, baselines, or controlled experiments.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

## Experimental Design Best Practice
1. ALWAYS include meaningful baselines (not just random):
   - At least one classical method baseline
   - At least one recent SOTA method baseline
   - A simple-but-strong baseline (e.g., linear probe, k-NN)
2. Use MULTIPLE random seeds (minimum 3, ideally 5)
3. Report mean +/- std across seeds
4. Design ablations that isolate EACH key component:
   - Remove one component at a time
   - Each ablation must be meaningfully different from baseline
5. Control variables: change only ONE thing per comparison
6. Use standard splits (train/val/test), never test on training data
7. Report wall-clock time and memory usage alongside accuracy