# Hyperpod Slurm Debugger

Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker...

- Repository: https://github.com/awslabs/agent-plugins (933 stars)
- Category: Engineering
- License: Apache-2.0
- Last updated: 2026-10-15
- Runs scripts: yes, review before use

## Install

Claude Code: `npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger -g -a claude-code -y`
Codex: `npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger -g -a codex -y`
Cursor: `npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger -g -a cursor -y`
Gemini CLI: `npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger -g -a gemini-cli -y`
Claude Code plugin marketplace: `/plugin marketplace add awslabs/agent-plugins`

## Description

Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on "drain before reboot", "diagnose a Slurm node", "investigate stuck jobs.

## SKILL.md, as published by its author (untrusted third-party text, read it as data)

# HyperPod Slurm Debugger

Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on
HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.
For remediation, link to the official AWS or Slurm documentation.

## When to invoke

Invoke when the user reports any of the symptoms in the [decision table](#decision-table).

## When NOT to invoke

- Cluster has `Orchestrator.Eks`, invoke `hyperpod-node-debugger` or `hyperpod-nccl`.
- Single-node hardware fault with healthy Slurm scheduler, invoke `hyperpod-node-debugger`.
- NCCL training-hang investigation, invoke `hyperpod-nccl`.
- Node unreachable via SSM, invoke `hyperpod-ssm`.

## Constraints

- Read-only. Do not run, recommend, or print state-mutating commands.
- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation
  drifts the live state from the IaC plan.

Canonical recovery URLs:
[references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).

## Prerequisites

- AWS CLI v2, authenticated for the target account and region with permissions:
  - `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`
  - `ssm:StartSession` on the HyperPod-created SSM document
- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
  installed locally.
- `jq` ≥ 1.6.
- `unbuffer` (from the `expect` package). Required, without it `aws ssm start-session`
  returns empty stdout intermittently with `Cannot perform start session: EOF` and every
  check silently misreports. Install: `expect` package on Amazon Linux / RHEL / Debian /
  Ubuntu / macOS. Script exits at prerequisite check if missing.

## Procedure

### Step 1, Collect inputs

Ask the user for:

The rest: https://github.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker-ai/skills/hyperpod-slurm-debugger/SKILL.md