What's New

Amazon SageMaker HyperPod now ships open-source "troubleshooting skills" that embed expert-level cluster diagnostics directly into AI coding assistants such as Claude Code, Cursor, and Kiro. These skills encode AWS best practices into structured diagnostic workflows, allowing operators to diagnose GPU faults, NCCL communication failures, software version drift, and performance bottlenecks through natural language rather than manual node inspection. The skills are available today for both Slurm and Amazon EKS-orchestrated HyperPod clusters via the SageMaker AI skills plugin on AWSLabs GitHub.

How It Works

• The skills are delivered as a plugin (sagemaker-ai) installed into a supported AI coding assistant; no modifications to existing HyperPod cluster infrastructure are required.
• Each skill is a structured diagnostic workflow that instructs the AI agent to collect evidence from cluster nodes using AWS Systems Manager (SSM), eliminating the need to manually SSH or SSM into individual instances.
• Thirteen discrete skills are provided, covering planning, model customization workflows, and four HyperPod-specific capabilities: hyperpod-ssm (remote command execution), hyperpod-version-checker (software version comparison across nodes), hyperpod-issue-report (automated diagnostic report generation), and hyperpod-cluster-debugger (deep cluster diagnostics).
• The AI agent follows the encoded workflows to systematically gather logs and metrics, analyze patterns across multiple nodes simultaneously, and surface actionable recommendations in natural language.
• Skills support both Slurm-orchestrated and Amazon EKS-orchestrated HyperPod clusters, covering the two primary orchestration models used in large-scale distributed training.
• The plugin is open source and hosted in the awslabs/agent-plugins GitHub repository, allowing teams to inspect, extend, or contribute to the diagnostic workflows.

Why It's Important

• Debugging distributed AI/ML clusters today requires deep expertise—operators must manually SSM into dozens of nodes, parse fragmented logs, and cross-reference documentation, making mean time to resolution (MTTR) high and costly.
• By encoding AWS best practices directly into the agent workflows, teams without deep HyperPod or GPU cluster expertise can perform expert-level diagnostics, democratizing operational knowledge.
• Natural language interaction removes the steep CLI and tooling learning curve, allowing ML engineers focused on model development to self-serve on infrastructure issues rather than escalating to specialized platform teams.
• Automated diagnostic reporting (hyperpod-issue-report) accelerates support case creation and reduces back-and-forth with AWS Support by providing structured, evidence-backed reports upfront.
• Faster cluster recovery directly translates to reduced wasted GPU-hours, which at the scale of foundation model training can represent significant cost savings.
• The open-source nature means organizations can customize diagnostic workflows to encode their own operational runbooks alongside AWS best practices.

How It's Different

• Unlike generic observability tools (e.g., CloudWatch dashboards or Prometheus), these skills are agentic—they actively drive a multi-step investigation workflow rather than passively presenting metrics for a human to interpret.
• Traditional runbooks and documentation require the operator to know what to look for; HyperPod skills guide the AI agent through systematic evidence collection, making the diagnostic process self-directing.
• The skills are tightly integrated with AWS Systems Manager for node access, meaning no additional network exposure, bastion hosts, or custom tooling is needed to reach cluster nodes.
• Compared to standalone monitoring solutions, the skills live inside the developer's existing coding assistant, keeping diagnostics in the same context as code and configuration work.
• The open-source, plugin-based architecture contrasts with proprietary observability platforms, giving teams full visibility into and control over the diagnostic logic.
• Coverage spans both Slurm and EKS orchestration in a single plugin, whereas most third-party tools require separate integrations or agents per orchestration layer.

When to Prefer It

• Use HyperPod troubleshooting skills when a distributed training job stalls or fails and the root cause is unclear—GPU hardware faults, NCCL collective communication errors, or node-level software mismatches are all covered.
• Prefer this approach when your team lacks dedicated MLOps or HPC infrastructure engineers and ML practitioners need to self-diagnose cluster issues without deep systems expertise.
• Ideal for organizations running large multi-node training jobs on HyperPod where manually inspecting dozens of instances is impractical within an acceptable recovery time window.
• Use the hyperpod-version-checker skill proactively after cluster updates or node replacements to detect software version drift before it causes training failures.
• Use hyperpod-issue-report when preparing to open an AWS Support case, as the automated report provides the structured evidence needed to accelerate triage.
• Teams already using Claude Code, Cursor, or Kiro as their primary development environment will benefit most, since the skills integrate directly into their existing workflow with no context switching.
• Prefer this over ad-hoc scripting when you want diagnostics that are repeatable, auditable, and aligned with AWS best practices rather than one-off investigative scripts.

Availability

• Status: Generally available (GA) as of June 1, 2026.
• Orchestration support: Available for both Slurm-orchestrated and Amazon EKS-orchestrated SageMaker HyperPod clusters.
• Supported coding assistants: Claude Code, Cursor, and Kiro are explicitly listed; any assistant supporting the plugin format (including .claude-plugin and .codex-plugin configurations visible in the repository) may be compatible.
• Distribution: Open source, available via the awslabs/agent-plugins GitHub repository; installed as the sagemaker-ai plugin into the coding assistant.
• Prerequisites: Existing HyperPod cluster with AWS Systems Manager agent accessible on nodes; no cluster-side modifications required.
• Pricing: The skills themselves are open source and carry no additional AWS charge; standard SageMaker HyperPod, AWS SSM, and coding assistant costs apply.
• Regional availability: Not explicitly stated in the announcement; availability is expected to follow existing SageMaker HyperPod regional footprint.

Related Resources

• https://github.com/awslabs/agent-plugins/tree/main/plugins/sagemaker-ai