Distributed Training
Multi-GPU and distributed training patterns with PyTorch DDP.
Install
npx skills add aiming-lab/AutoResearchClaw --skill distributed-training -g -a claude-code -y- Repository
- aiming-lab/AutoResearchClaw14.8k
- Category
- Engineering
- Updated
- 3mo ago
- License
- MIT
- Context cost
- Lightabout 109 tokens
- Runs scripts
- No, instructions only
What it does
Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs.
Distributed Training Best Practice
- Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
- Initialize process group: dist.init_process_group(backend='nccl')
- Use DistributedSampler for data sharding
- Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
- Only save checkpoint on rank 0
- Scale learning rate linearly with world size
- Use gradient accumulation for effectively larger batch sizes
