# Hyperpod Performance Debugger

Diagnose performance issues on Amazon SageMaker HyperPod clusters, uneven NCCL...

- Repository: https://github.com/awslabs/agent-plugins (933 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-15
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger -g -a claude-code -y`
Codex: `npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger -g -a codex -y`
Cursor: `npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger -g -a cursor -y`
Gemini CLI: `npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add awslabs/agent-plugins`

## Description

Diagnose performance issues on Amazon SageMaker HyperPod clusters, uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# HyperPod Performance Debugger

1. **Uneven NCCL performance across nodes**, workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
2. **Poor filesystem performance**, training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.

## Scope and delegation

Route findings outside the two in-scope scenarios to the owner skill below.

| Concern observed                                                       | Route to                                                     |
| ---------------------------------------------------------------------- | ------------------------------------------------------------ |
| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace  | `hyperpod-node-debugger` (§ F Hardware/Auto-Repair, § G GPU) |
| `Cannot allocate memory` at `os.fork()`, root volume exhausted         | `hyperpod-node-debugger` (§ I Resource Exhaustion)           |
| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | `hyperpod-nccl`                                              |
| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes           | `hyperpod-version-checker`                                   |
| EFA self-referencing security-group rule missing, single node         | `hyperpod-node-debugger` § A (EFA / Security Group)          |
| EFA self-referencing security-group rule missing, cluster-wide        | `hyperpod-cluster-debugger` § A (EFA Health Checks)          |
| Slurm node state changes (drain / resume / reboot)                     | `hyperpod-slurm-debugger`                                    |
| Diagnostic bundle for AWS Support                                      | `hyperpod-issue-report`                                      |
| Shell access on a node                                                 | `hyperpod-ssm`                                               |

## Operating policy

The rest: https://github.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker-ai/skills/hyperpod-performance-debugger/SKILL.md