Ml Training Recipes
Battle-tested PyTorch training recipes for all domains, LLMs, vision, diffusion...
Install
npx skills add Orchestra-Research/AI-Research-SKILLs --skill ml-training-recipes -g -a claude-code -y- Repository
- Orchestra-Research/AI-Research-SKILLs14.2k
- Category
- Testing
- Updated
- 25d ago
- License
- MIT
- Context cost
- Mediumabout 2,690 tokens
- Runs scripts
- No, instructions only
What it does
Battle-tested PyTorch training recipes for all domains, LLMs, vision, diffusion, medical imaging, protein/drug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.
ML Training Recipes
Battle-tested patterns for PyTorch training across domains. Drawn from production codebases (Karpathy's autoresearch/nanochat, torchvision, HuggingFace) and modern training practice.
Reference files (read when needed)
references/architecture.md, Transformer/LLM architecture code patterns, weight initreferences/optimizers.md, Muon, AdamW hybrid, per-group LR, compiled optimizer stepsreferences/domain-specific.md, Vision, diffusion, contrastive, distributed, checkpointing, data loadingreferences/scaling-and-selection.md, Scaling laws, compute budget tables, decision trees, DGX Sparkreferences/biomedical.md, Drug discovery, protein models, medical imaging, genomics, clinical NLPreferences/experiment-loop.md, Autonomous experiment loop (autoresearch keep/discard/revert)
Architecture Selection
Pick the right model by data type and data scale:
| Data Type | < 10K samples | 10K-100K | > 100K |
|---|---|---|---|
| Images | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |
| Text (gen) | Few-shot prompting | Fine-tune GPT/LLaMA (LoRA) | Pretrain from scratch |
| Tabular | XGBoost/LightGBM | Still XGBoost | Neural viable |
| Audio | Pretrained Whisper | Fine-tune AST | Train from scratch |
| Molecules | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |
| Proteins | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |
| Medical img | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR / MedSAM |
Key principle: architecture matters less than training recipe at equal compute. A well-tuned ResNet beats a poorly-tuned ViT (ref: "ResNet Strikes Back", Wightman 2021).
For biomedical domains, see references/biomedical.md.
For sequence model selection and compute planning, see references/scaling-and-selection.md.
