Hyperpod Slurm Debugger
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker...
Install
npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger -g -a claude-code -y- Repository
- awslabs/agent-plugins933
- Category
- Engineering
- Updated
- 20d ago
- License
- Apache-2.0
- Context cost
- Mediumabout 3,000 tokens
- Runs scripts
- Yes, review before use
What it does
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on "drain before reboot", "diagnose a Slurm node", "investigate stuck jobs.
HyperPod Slurm Debugger
Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command. For remediation, link to the official AWS or Slurm documentation.
When to invoke
Invoke when the user reports any of the symptoms in the decision table.
When NOT to invoke
- Cluster has
Orchestrator.Eks, invokehyperpod-node-debuggerorhyperpod-nccl. - Single-node hardware fault with healthy Slurm scheduler, invoke
hyperpod-node-debugger. - NCCL training-hang investigation, invoke
hyperpod-nccl. - Node unreachable via SSM, invoke
hyperpod-ssm.
Constraints
- Read-only. Do not run, recommend, or print state-mutating commands.
- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation drifts the live state from the IaC plan.
Canonical recovery URLs: references/slurm-details.md → Authoritative recovery documentation.
Prerequisites
- AWS CLI v2, authenticated for the target account and region with permissions:
sagemaker:DescribeCluster,sagemaker:ListClusterNodesssm:StartSessionon the HyperPod-created SSM document
- Session Manager plugin installed locally.
jq≥ 1.6.unbuffer(from theexpectpackage). Required, without itaws ssm start-sessionreturns empty stdout intermittently withCannot perform start session: EOFand every check silently misreports. Install:expectpackage on Amazon Linux / RHEL / Debian / Ubuntu / macOS. Script exits at prerequisite check if missing.
Procedure
Step 1, Collect inputs
Ask the user for:
