# Distributed Training

Multi-GPU and distributed training patterns with PyTorch DDP.

- Repository: https://github.com/aiming-lab/AutoResearchClaw (14798 stars)
- Category: Engineering
- License: MIT
- Last updated: 2026-08-19
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add aiming-lab/AutoResearchClaw --skill distributed-training -g -a claude-code -y`
Codex: `npx skills add aiming-lab/AutoResearchClaw --skill distributed-training -g -a codex -y`
Cursor: `npx skills add aiming-lab/AutoResearchClaw --skill distributed-training -g -a cursor -y`
Gemini CLI: `npx skills add aiming-lab/AutoResearchClaw --skill distributed-training -g -a gemini-cli -y`

## Description

Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

## Distributed Training Best Practice
1. Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
2. Initialize process group: dist.init_process_group(backend='nccl')
3. Use DistributedSampler for data sharding
4. Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
5. Only save checkpoint on rank 0
6. Scale learning rate linearly with world size
7. Use gradient accumulation for effectively larger batch sizes