# Nanogpt

Educational GPT implementation in ~300 lines.

- Repository: https://github.com/Orchestra-Research/AI-Research-SKILLs (14176 stars)
- Category: Engineering
- License: MIT
- Last updated: 2026-10-11
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill nanogpt -g -a claude-code -y`
Codex: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill nanogpt -g -a codex -y`
Cursor: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill nanogpt -g -a cursor -y`
Gemini CLI: `npx skills add Orchestra-Research/AI-Research-SKILLs --skill nanogpt -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add Orchestra-Research/AI-Research-SKILLs`

## Description

Educational GPT implementation in ~300 lines. Reproduces GPT-2 (124M) on OpenWebText. Clean, hackable code for learning transformers. By Andrej Karpathy. Perfect for understanding GPT architecture from scratch. Train on Shakespeare (CPU) or OpenWebText (multi-GPU).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# nanoGPT - Minimalist GPT Training

## Quick start

nanoGPT is a simplified GPT implementation designed for learning and experimentation.

**Installation**:
```bash
pip install torch numpy transformers datasets tiktoken wandb tqdm
```

**Train on Shakespeare** (CPU-friendly):
```bash
# Prepare data
python data/shakespeare_char/prepare.py

# Train (5 minutes on CPU)
python train.py config/train_shakespeare_char.py

# Generate text
python sample.py --out_dir=out-shakespeare-char
```

**Output**:
```
ROMEO:
What say'st thou? Shall I speak, and be a man?

JULIET:
I am afeard, and yet I'll speak; for thou art
One that hath been a man, and yet I know not
What thou art.
```

## Common workflows

### Workflow 1: Character-level Shakespeare

**Complete training pipeline**:
```bash
# Step 1: Prepare data (creates train.bin, val.bin)
python data/shakespeare_char/prepare.py

# Step 2: Train small model
python train.py config/train_shakespeare_char.py

# Step 3: Generate text
python sample.py --out_dir=out-shakespeare-char
```

**Config** (`config/train_shakespeare_char.py`):
```python
# Model config
n_layer = 6          # 6 transformer layers
n_head = 6           # 6 attention heads
n_embd = 384         # 384-dim embeddings
block_size = 256     # 256 char context

# Training config
batch_size = 64
learning_rate = 1e-3
max_iters = 5000
eval_interval = 500

# Hardware
device = 'cpu'  # Or 'cuda'
compile = False # Set True for PyTorch 2.0
```

**Training time**: ~5 minutes (CPU), ~1 minute (GPU)

### Workflow 2: Reproduce GPT-2 (124M)

**Multi-GPU training on OpenWebText**:
```bash
# Step 1: Prepare OpenWebText (takes ~1 hour)
python data/openwebtext/prepare.py

# Step 2: Train GPT-2 124M with DDP (8 GPUs)
torchrun --standalone --nproc_per_node=8 \
  train.py config/train_gpt2.py

# Step 3: Sample from trained model
python sample.py --out_dir=out
```

The rest: https://github.com/Orchestra-Research/AI-Research-SKILLs/blob/HEAD/01-model-architecture/nanogpt/SKILL.md