Hyperpod Performance Debugger
Diagnose performance issues on Amazon SageMaker HyperPod clusters, uneven NCCL...
Install
npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger -g -a claude-code -y- Repository
- awslabs/agent-plugins933
- Category
- Engineering
- Updated
- 20d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- Yes, review before use
What it does
Diagnose performance issues on Amazon SageMaker HyperPod clusters, uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.
HyperPod Performance Debugger
- Uneven NCCL performance across nodes, workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
- Poor filesystem performance, training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.
Scope and delegation
Route findings outside the two in-scope scenarios to the owner skill below.
| Concern observed | Route to |
|---|---|
| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace | hyperpod-node-debugger (§ F Hardware/Auto-Repair, § G GPU) |
Cannot allocate memory at os.fork(), root volume exhausted | hyperpod-node-debugger (§ I Resource Exhaustion) |
| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | hyperpod-nccl |
| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes | hyperpod-version-checker |
| EFA self-referencing security-group rule missing, single node | hyperpod-node-debugger § A (EFA / Security Group) |
| EFA self-referencing security-group rule missing, cluster-wide | hyperpod-cluster-debugger § A (EFA Health Checks) |
| Slurm node state changes (drain / resume / reboot) | hyperpod-slurm-debugger |
| Diagnostic bundle for AWS Support | hyperpod-issue-report |
| Shell access on a node | hyperpod-ssm |
