Data Loading
Optimize data loading pipeline to prevent GPU starvation.
Install
npx skills add aiming-lab/AutoResearchClaw --skill data-loading -g -a claude-code -y- Repository
- aiming-lab/AutoResearchClaw14.8k
- Category
- Engineering
- Updated
- 3mo ago
- License
- MIT
- Context cost
- Lightabout 108 tokens
- Runs scripts
- No, instructions only
What it does
Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.
Efficient Data Loading Best Practice
- Use num_workers = min(8, os.cpu_count()) for DataLoader
- Enable pin_memory=True when using GPU
- Use persistent_workers=True to avoid re-spawning
- Pre-compute and cache transformations when possible
- For image data: use torchvision.transforms.v2 (faster)
- For large datasets: consider memory-mapped files or WebDataset
- Profile with torch.utils.bottleneck to find I/O bottlenecks
