HyperPod Issue Report

Collect diagnostic logs from HyperPod cluster nodes via SSM, store results in S3. Supports both EKS and Slurm clusters with auto-detection. Uses the bundled scripts/hyperpod_issue_report.py for reliable parallel collection.

Prerequisites

• AWS CLI configured with permissions: sagemaker:DescribeCluster, sagemaker:ListClusterNodes, ssm:StartSession, s3:PutObject, s3:GetObject, eks:DescribeCluster
• Python 3.8+ and uv (see uv installation docs for install options)
• SSM Agent running on target nodes; node IAM roles need s3:GetObject/s3:PutObject on the report bucket
• For EKS clusters: kubectl installed and configured (see Workflow step 2)

Workflow

1. Gather Information

Collect from the user:

• Cluster identifier (required): accepts cluster name or full cluster ARN (e.g., arn:aws:sagemaker:us-west-2:123456789012:cluster/abc123)
• AWS region (required unless extractable from ARN)
• S3 path for report storage (required, e.g. s3://bucket/prefix). If the user doesn't have a bucket, create one (e.g., s3://hyperpod-diagnostics-<account-id>-<region>)
• Issue description (optional)
• Target scope: all nodes, specific instance groups, or specific node IDs (optional)

2. Verify Environment

aws sts get-caller-identity
aws sagemaker describe-cluster --cluster-name <name-or-arn> --region <region>

If the S3 bucket doesn't exist, create it:

aws s3 mb s3://<bucket-name> --region <region>

For EKS clusters (check Orchestrator.Eks in describe-cluster output):

1. Ensure kubectl is installed (which kubectl). If missing, install it for the current platform.

2. Configure kubeconfig using the EKS cluster name from the describe-cluster response:

   aws eks update-kubeconfig --name <eks-cluster-name> --region <region>
   

3. Run the Collection Script