Experimental Design
Best practices for designing reproducible ML experiments.
Install
npx skills add aiming-lab/AutoResearchClaw --skill experimental-design -g -a claude-code -y- Repository
- aiming-lab/AutoResearchClaw14.8k
- Category
- Design
- Updated
- 3mo ago
- License
- MIT
- Context cost
- Lightabout 170 tokens
- Runs scripts
- No, instructions only
What it does
Best practices for designing reproducible ML experiments. Use when planning ablations, baselines, or controlled experiments.
Experimental Design Best Practice
- ALWAYS include meaningful baselines (not just random):
- At least one classical method baseline
- At least one recent SOTA method baseline
- A simple-but-strong baseline (e.g., linear probe, k-NN)
- Use MULTIPLE random seeds (minimum 3, ideally 5)
- Report mean +/- std across seeds
- Design ablations that isolate EACH key component:
- Remove one component at a time
- Each ablation must be meaningfully different from baseline
- Control variables: change only ONE thing per comparison
- Use standard splits (train/val/test), never test on training data
- Report wall-clock time and memory usage alongside accuracy
