# Large Model Training Guide

Gives methods for training, scaling and recovering large language and vision models.

- Repository: https://github.com/stas00/ml-engineering (19644 stars)
- Category: Data and ML
- License: CC-BY-SA-4.0
- Last updated: 2026-10-27
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add stas00/ml-engineering --skill ml-engineering -g -a claude-code -y`
Codex: `npx skills add stas00/ml-engineering --skill ml-engineering -g -a codex -y`
Cursor: `npx skills add stas00/ml-engineering --skill ml-engineering -g -a cursor -y`
Gemini CLI: `npx skills add stas00/ml-engineering --skill ml-engineering -g -a gemini-cli -y`

## Description

Field-tested methodology and concrete recipes for training and operating large-scale LLM/VLM/multi-modal models end to end - choosing and benchmarking accelerators, storage and network; SLURM/Kubernetes orchestration; maximizing training throughput and fitting models in memory; diagnosing and surviving training instabilities, NaN/Inf, and hardware/job failures; checkpointing and fault tolerance; inference performance and memory; debugging multi-node/ multi-GPU hangs; and writing/running tests. Use when the user is training or fine-tuning large models, hits low TFLOPS/MFU, OOM, slow dataloading, a loss spike/divergence, a NCCL/InfiniBand or multi-node hang, node/GPU failures, checkpoint or preemption problems, storage/network bottlenecks, or needs to pick GPUs/cloud/file-systems or size inference latency/throughput. Distilled from "Machine Learning Engineering", the latest version of which can be found at https://github.com/stas00/ml-engineering The latest SKILL.md version can be found at https://github.com/stas00/ml-engineering/blob/master/skills/ml-engineering/SKILL.md

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# Machine Learning Engineering

Distilled from **Machine Learning Engineering Open Book** by Stas Bekman - source: https://github.com/stas00/ml-engineering (CC BY-SA 4.0). Know-how from training BLOOM-176B, IDEFICS-80B and production RAG and RL training and inference systems. This skill is a condensed index; each section links back to the full chapter for depth, scripts, and benchmarks.

A field-tested, end-to-end guide to **training and serving large models** (LLMs, VLMs, multi-modal, RAG) on real hardware at scale - distilled from actually training BLOOM-176B and IDEFICS-80B and building production inference/RAG systems. It is a practitioner's brain dump: opinionated guidance backed by copy-paste scripts, benchmark tools, and comparison tables, written for the engineers and operators who have to make expensive clusters actually deliver a finished model.

It spans the entire stack that decides whether a run succeeds and how much it costs: selecting and *benchmarking* accelerators, storage, and network so the fast compute is never starved; orchestrating jobs with SLURM/Kubernetes; maximizing throughput (MFU) and fitting models in memory via parallelism (DP/TP/PP/ZeRO), activation recomputation, and offload; keeping training numerically stable through loss spikes and NaN/Inf; and surviving the *inevitable* hardware and job failures with frequent checkpointing, spare capacity, and automatic restarts. On the serving side it covers inference latency/throughput/cost trade-offs, KV-cache and memory sizing, and framework selection - plus diagnosing multi-node/multi-GPU hangs and testing the whole thing.

Use it as an operator's runbook: figure out which resource is *actually* the bottleneck (compute? memory? network? storage? dataloader?), then jump to the targeted recipe. For pure debugging technique (gdb/strace/py-spy/CUDA), pair this with [The Art of Debugging](https://github.com/stas00/the-art-of-debugging/blob/master/SKILL.md).

## Core principles

The rest: https://github.com/stas00/ml-engineering/blob/HEAD/skills/ml-engineering/SKILL.md