# Ml Training Recipes

Battle-tested PyTorch training recipes for all domains, LLMs, vision, diffusion...

- Repository: https://github.com/Orchestra-Research/AI-Research-SKILLs (14176 stars)
- Category: Testing
- License: MIT
- Last updated: 2026-10-11
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill ml-training-recipes -g -a claude-code -y`
Codex: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill ml-training-recipes -g -a codex -y`
Cursor: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill ml-training-recipes -g -a cursor -y`
Gemini CLI: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill ml-training-recipes -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`

## Description

Battle-tested PyTorch training recipes for all domains, LLMs, vision, diffusion, medical imaging, protein/drug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# ML Training Recipes

Battle-tested patterns for PyTorch training across domains. Drawn from production codebases
(Karpathy's autoresearch/nanochat, torchvision, HuggingFace) and modern training practice.

## Reference files (read when needed)

- `references/architecture.md`, Transformer/LLM architecture code patterns, weight init
- `references/optimizers.md`, Muon, AdamW hybrid, per-group LR, compiled optimizer steps
- `references/domain-specific.md`, Vision, diffusion, contrastive, distributed, checkpointing, data loading
- `references/scaling-and-selection.md`, Scaling laws, compute budget tables, decision trees, DGX Spark
- `references/biomedical.md`, Drug discovery, protein models, medical imaging, genomics, clinical NLP
- `references/experiment-loop.md`, Autonomous experiment loop (autoresearch keep/discard/revert)

---

## Architecture Selection

Pick the right model by **data type** and **data scale**:

| Data Type | < 10K samples | 10K-100K | > 100K |
|-----------|--------------|----------|--------|
| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |
| **Text (gen)** | Few-shot prompting | Fine-tune GPT/LLaMA (LoRA) | Pretrain from scratch |
| **Tabular** | XGBoost/LightGBM | Still XGBoost | Neural viable |
| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |
| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |
| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |
| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR / MedSAM |

**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned
ResNet beats a poorly-tuned ViT (ref: "ResNet Strikes Back", Wightman 2021).

For biomedical domains, see `references/biomedical.md`.
For sequence model selection and compute planning, see `references/scaling-and-selection.md`.

---

## Scaling Laws

### Chinchilla

The rest: https://github.com/Orchestra-Research/AI-Research-SKILLs/blob/HEAD/10-optimization/ml-training-recipes/SKILL.md