Amazon SageMaker HyperPod now supports deep health checks for Slurm clusters with continuous provisioning
Validate every GPU node before it runs a job — deep health checks now work with async continuous provisioning to prevent silent hardware failures.
View original announcement →Visual Summary
graph TD
A{{HyperPod Deep Health Checks}}:::announced
B(Amazon SageMaker HyperPod):::compute
C([Continuous Provisioning]):::feature
D([Instance Stress Tests]):::feature
E([Instance Connectivity Tests]):::feature
F([Auto Node Recovery]):::feature
G(Slurm Scheduler):::compute
H((ML Engineers)):::external
H ==>|"triggers"| A
A ==>|"validates nodes"| B
B -->|"scales async"| C
C -.->|"new nodes"| A
A -->|"GPU/CPU/EFA"| D
A -->|"NCCL multi-node"| E
A -->|"isolates nodes"| G
A -.->|"failed nodes"| F
F -->|"replaces/reboots"| B
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon SageMaker HyperPod now supports deep health checks for Slurm-orchestrated clusters that use continuous provisioning, allowing teams to proactively validate GPU accelerator health on running instances at any time. The feature enables comprehensive hardware stress tests and inter-node connectivity tests to be run on individual nodes or entire instance groups before workloads are scheduled on them. This pairs the asynchronous scaling flexibility of continuous provisioning with rigorous hardware validation, closing a critical gap where a single faulty node could silently waste hours of expensive compute time.
How It Works
- Two check categories are supported:
InstanceStressruns instance-level tests including CPU/memory/disk stress, GPU/PCI verification, DCGM GPU diagnostics at level 4 (covering additional memory tests, ~45–90 minutes depending on GPU count), and EFA loopback bandwidth/latency tests.InstanceConnectivityruns multi-node NCCL tests to validate inter-node GPU communication performance across the cluster. - Triggering mechanisms: Checks can run automatically when instances are created or added to a cluster (
on-start), or they can be triggered manually at any time via theStartClusterHealthCheckAPI (on-demand), giving operators full control over validation timing. - Automatic workload isolation: Nodes undergoing checks are placed into a Slurm maintenance reservation, preventing the scheduler from assigning jobs to them. Once all checks pass, nodes are automatically released back into the available pool without manual intervention.
- Granular targeting: Operators can scope a health check to an entire instance group or to specific individual instances, enabling surgical validation of newly provisioned nodes without disturbing healthy, already-running workloads.
- Visibility and observability: Progress and results are surfaced at both the instance group and individual instance level through the SageMaker console and APIs, covering GPU health, EFA network connectivity, and multi-node NCCL communication metrics.
- Integrated recovery loop: When a node fails a deep health check, HyperPod's existing automatic node recovery capability kicks in to reboot or replace the failed instance, creating a closed-loop remediation workflow.
- AMI prerequisite: The feature requires the latest HyperPod AMI version; operators must run
UpdateClusterSoftwarebefore enabling deep health checks on existing clusters.
Why It's Important
- Eliminates silent compute waste: A single unhealthy GPU node in a distributed training job can corrupt results or stall the entire run for hours before the problem is detected; proactive validation catches faults before they consume expensive accelerator time.
- Closes the continuous-provisioning gap: Continuous provisioning's asynchronous node addition previously meant new nodes could enter the cluster without full hardware validation; deep health checks now fill that gap, ensuring every node is verified before it receives work.
- Reduces mean time to recovery: By combining automated detection, isolation, and HyperPod's node replacement, the time between a hardware fault occurring and a healthy replacement being in service is dramatically shortened compared to manual triage workflows.
- Protects long-running training jobs: Healthy nodes already running jobs are never interrupted during checks on newly added nodes, preserving training progress and SLA commitments.
- Improves cluster-wide confidence: Teams can validate GPU health, EFA fabric integrity, and NCCL collective performance in a single workflow, giving a holistic picture of cluster readiness before launching large-scale foundation model or HPC workloads.
How It's Different
- On-demand vs. only on-start validation: Many managed cluster services only validate hardware at instance launch; HyperPod now allows health checks to be triggered at any point in the cluster lifecycle, catching hardware degradation that develops over time.
- Continuous-provisioning awareness: Unlike static cluster validation, this feature is designed for asynchronous node arrival — each node can be checked independently as it comes online, without requiring the full cluster to be present or pausing other work.
- Closed-loop remediation: The integration with HyperPod's automatic node recovery means detection and replacement are automated end-to-end, rather than simply alerting operators to a problem and requiring manual action.
- Slurm-native isolation mechanism: Isolation is implemented via Slurm maintenance reservations, meaning it works within the existing scheduler's semantics rather than requiring a separate orchestration layer or custom drain scripts.
- Multi-layer validation in one API call: A single
StartClusterHealthCheckinvocation can cover CPU, memory, disk, GPU/NVLink counts, DCGM level-4 diagnostics, EFA loopback, and NCCL multi-node tests — breadth that typically requires assembling multiple independent tools. - Granular scope control: Operators can target a single suspect instance rather than taking an entire instance group offline, minimizing disruption compared to cluster-wide maintenance windows.
When to Prefer It
- Scaling out a cluster mid-training: When continuous provisioning adds new GPU nodes to an active Slurm cluster, run on-demand deep health checks on each new node before the scheduler assigns distributed training tasks to it.
- Pre-flight validation for large foundation model runs: Before launching a multi-week, multi-node LLM pre-training job where a hardware fault mid-run would be extremely costly, use deep health checks to certify every node in the target instance group.
- Post-maintenance or post-reboot verification: After a node has been rebooted due to a prior failure or OS update, trigger an on-demand check to confirm hardware is fully healthy before returning it to the job queue.
- Periodic cluster health audits: For long-lived clusters where GPU hardware can degrade over time (e.g., thermal issues, memory errors), schedule regular on-demand checks to catch silent degradation before it impacts production workloads.
- Debugging intermittent job failures: When distributed training jobs fail with non-deterministic errors that suggest hardware issues, use targeted instance-level checks to isolate the faulty node without taking the entire cluster offline.
- Capacity validation after spot or on-demand instance replacement: When HyperPod automatically replaces a failed node with a new instance, use on-start or on-demand checks to confirm the replacement hardware is fully functional before trusting it with critical workloads.
Availability
- General Availability: This feature is generally available as of July 9, 2026; it is not in preview.
- Regional availability: Available in all AWS regions where Amazon SageMaker HyperPod is currently supported.
- Cluster type scope: Applies specifically to Slurm-orchestrated HyperPod clusters created with continuous provisioning; EKS-orchestrated clusters and static-provisioning Slurm clusters are not covered by this announcement.
- AMI requirement: Clusters must be running the latest HyperPod AMI version; existing clusters require a
UpdateClusterSoftwareoperation before the feature can be used. - Instance type limitation: The
InstanceConnectivity(NCCL) check is only supported on instance types with multi-node GPU communication capabilities; single-GPU or CPU-only instances are limited toInstanceStresschecks. - Pricing: No separate charge is listed for the health check feature itself; standard SageMaker HyperPod instance and compute costs apply while checks are running.
