# Hyperpod Nccl

Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS...

- Repository: https://github.com/awslabs/agent-plugins (933 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-15
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add awslabs/agent-plugins --skill hyperpod-nccl -g -a claude-code -y`
Codex: `npx skills add awslabs/agent-plugins --skill hyperpod-nccl -g -a codex -y`
Cursor: `npx skills add awslabs/agent-plugins --skill hyperpod-nccl -g -a cursor -y`
Gemini CLI: `npx skills add awslabs/agent-plugins --skill hyperpod-nccl -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add awslabs/agent-plugins`

## Description

Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm), training hangs, AllReduce / collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, /dev/shm or memlock issues, NCCL version mismatch across pods, container OOM / exit-137 / OOMKilled, GPU OOM (CUDA out of memory), CrashLoopBackOff / Pending pods, MASTER_ADDR DNS, NetworkPolicy blocking. Not for single-node hardware faults (→ hyperpod-node-debugger § G) or cluster-creation EFA / SSM failures (→ hyperpod-cluster-debugger § A / § F).

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# HyperPod NCCL Debugger

**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state, present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.

Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). `scripts/nccl-diagnose.sh` reads state via AWS APIs, kubectl, and SSM, then prints each issue as `[FAIL] ... → references/<file>.md § <section>`. Read-only.

**Signal sourcing:** `list-cluster-events` carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does **not** carry NCCL timeouts, GPU XID/ECC, or per-pod training signals, those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health.

---

## Workflow

1. Collect cluster name, region, namespace/job (EKS), exact NCCL error string.
2. Run the diagnostic (always, the output drives everything else).
3. For every `[FAIL]` line, `Read` the referenced section.
4. Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval.
5. Re-run the diagnostic to confirm.

If a finding has no matching section, report it as a bug, do not invent a fix.

## Step 1: Authenticate kubectl (EKS)

```bash
EKS_ARN=$(aws sagemaker describe-cluster --cluster-name <HYPERPOD-NAME> --region <REGION> \
  --query 'Orchestrator.Eks.ClusterArn' --output text)
EKS_NAME=$(echo "$EKS_ARN" | awk -F'/' '{print $NF}')
aws eks update-kubeconfig --name "$EKS_NAME" --region <REGION>
kubectl get nodes
```

## Step 2: Run the diagnostic

The rest: https://github.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker-ai/skills/hyperpod-nccl/SKILL.md