# Data Loading

Optimize data loading pipeline to prevent GPU starvation.

- Repository: https://github.com/aiming-lab/AutoResearchClaw (14798 stars)
- Category: Engineering
- License: MIT
- Last updated: 2026-08-19
- Runs scripts: no, instructions only

## Install

Claude Code: `npx skills add aiming-lab/AutoResearchClaw --skill data-loading -g -a claude-code -y`
Codex: `npx skills add aiming-lab/AutoResearchClaw --skill data-loading -g -a codex -y`
Cursor: `npx skills add aiming-lab/AutoResearchClaw --skill data-loading -g -a cursor -y`
Gemini CLI: `npx skills add aiming-lab/AutoResearchClaw --skill data-loading -g -a gemini-cli -y`

## Description

Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

## Efficient Data Loading Best Practice
1. Use num_workers = min(8, os.cpu_count()) for DataLoader
2. Enable pin_memory=True when using GPU
3. Use persistent_workers=True to avoid re-spawning
4. Pre-compute and cache transformations when possible
5. For image data: use torchvision.transforms.v2 (faster)
6. For large datasets: consider memory-mapped files or WebDataset
7. Profile with torch.utils.bottleneck to find I/O bottlenecks