What's New

Amazon SageMaker HyperPod now provides AMI versioning and auto-patching capabilities for clusters orchestrated by Amazon EKS, giving administrators full visibility into the exact AMI version running on every node and instance group. AMI versions are expressed in semantic versioning (major.minor.patch) format, and auto-patching can be enabled per instance group to automatically apply backward-compatible security patches as nodes become idle. A new AMI support policy also formalizes support timelines for each AMI version, clarifying when AWS will stop publishing security patches for a given release.

How It Works

• AMI versioning: Every instance group and individual node in a HyperPod cluster now exposes its AMI version in major.minor.patch semantic format, making it straightforward to detect version drift across a heterogeneous cluster.
• Rollback via UpdateClusterSoftware API: Administrators can roll back to a prior AMI version—including its specific NVIDIA driver, CUDA, and OS stack—by specifying a target ImageReleaseVersion per instance group, with optional DeploymentConfig settings for rolling updates and auto-rollback via CloudWatch alarms.
• Auto-patching (AutoPatchConfig): Configured at the instance-group level through CreateCluster or UpdateCluster, auto-patching applies only patch-level (backward-compatible) security fixes. It never increments the major or minor version of bundled components such as NVIDIA drivers, CUDA, or the OS kernel.
• Idle-node targeting: Patches are applied only when a node becomes idle, ensuring that long-running, multi-day training jobs are not interrupted mid-execution.
• Patching strategy controls: The AutoPatchConfig object supports PatchingStrategy, PatchSchedule (including NextPatchDate), and a DeploymentConfig with rolling-update batch sizes and wait intervals, giving fine-grained control over rollout behavior.
• AMI support policy: A published policy defines support windows—12 months for major versions, 6 months for minor versions, and patch versions are supported until a newer patch is released—after which AWS stops shipping security patches for that version.
• Scope of patched components: The support policy and patching mechanism cover EFA, NVIDIA Driver, NCCL (aws-ofi-nccl), CUDA, and the OS kernel.

Why It's Important

• Eliminates security blind spots: Previously, administrators had no standardized way to know which AMI version was running on each node, making it nearly impossible to audit security posture or detect drift at scale.
• Solves the long-running job patching dilemma: Multi-day or multi-week foundation model training jobs could not be safely interrupted for patching; idle-node targeting resolves this by waiting for natural job completion before applying patches.
• Prevents accidental software stack changes: Manual patching risked inadvertently upgrading NVIDIA drivers or CUDA to incompatible versions, potentially breaking distributed training workloads; the patch-only constraint eliminates this risk.
• Reduces operational burden: Auto-patching removes the need for a dedicated ops workflow to monitor CVEs, schedule maintenance windows, and manually apply patches across potentially hundreds of GPU nodes.
• Enables safe rollback: The ability to revert to a prior AMI version—including its full software stack—provides a safety net when a patch introduces unexpected behavior, reducing mean time to recovery.
• Formalizes lifecycle planning: The published AMI support policy lets platform teams plan upgrades proactively rather than reactively, aligning cluster maintenance with known end-of-support dates.

How It's Different

• Semantic versioning vs. opaque AMI IDs: Traditional EC2 AMI management relies on opaque AMI IDs with no inherent version semantics; HyperPod now surfaces structured major.minor.patch versions natively, making drift detection and policy enforcement straightforward.
• Workload-aware patching vs. scheduled maintenance windows: Standard patching approaches (e.g., AWS Systems Manager Patch Manager) apply patches on a time-based schedule regardless of workload state; HyperPod auto-patching waits for node idleness, making it purpose-built for long-running AI/ML jobs.
• Patch-only constraint vs. unconstrained updates: Unlike general-purpose patching tools that may upgrade any package, HyperPod auto-patching is explicitly constrained to patch-level changes, guaranteeing that NVIDIA driver, CUDA, and kernel major/minor versions remain stable.
• Per-instance-group granularity: Patching can be enabled or disabled independently for each instance group within a cluster, allowing different policies for head nodes, training nodes, and inference nodes within the same cluster.
• Integrated rollback with full stack restoration: Rolling back via UpdateClusterSoftware restores the entire bundled software stack (NVIDIA driver, CUDA, EFA, NCCL) to the prior version, not just the OS image, which is critical for reproducibility in AI/ML workloads.
• Published support policy: Unlike ad-hoc AMI releases, the formal support policy with defined windows (12/6 months for major/minor) provides predictability that is absent from standard EC2 AMI management.

When to Prefer It

• Multi-day or multi-week foundation model pre-training runs: When training jobs cannot tolerate interruption, auto-patching ensures security patches are applied only when nodes are naturally idle between jobs.
• Large clusters with many instance groups: When managing dozens or hundreds of nodes across multiple instance groups, AMI versioning provides the audit trail and drift detection needed to maintain a consistent, compliant security posture.
• Regulated or compliance-sensitive environments: Organizations subject to SOC 2, FedRAMP, or internal security SLAs that require timely CVE remediation can use auto-patching to meet patch SLAs without manual intervention.
• Teams that need reproducible training environments: When experiment reproducibility requires pinning specific NVIDIA driver and CUDA versions, AMI versioning and the rollback capability ensure the exact software stack can be restored if a patch causes issues.
• Platform teams managing shared HyperPod clusters: Centralized ML platform teams supporting multiple data science teams benefit from the per-instance-group patching controls and the ability to enforce a consistent AMI version policy across the cluster.
• Organizations planning long-term cluster lifecycles: Teams that need to plan infrastructure upgrades 6–12 months in advance can use the AMI support policy to schedule major/minor version upgrades before end-of-support dates.

Availability

• General Availability: Both AMI versioning and auto-patching are generally available as of July 2, 2026.
• Orchestration scope: Currently available only for HyperPod clusters orchestrated by Amazon EKS; Slurm-orchestrated clusters are not yet supported.
• Regional availability: Available in all AWS Regions where Amazon SageMaker HyperPod is supported.
• API surface: Features are accessible via the UpdateClusterSoftware, CreateCluster, and UpdateCluster APIs, as well as through the AWS Management Console and CLI.
• Pricing: No additional charge for AMI versioning or auto-patching; standard SageMaker HyperPod instance pricing applies.
• Custom AMI support: Custom AMIs are supported with UpdateClusterSoftware, but must be unencrypted, owned by the same AWS account, and only the primary AMI snapshot volume is supported (additional volumes are not).
• AMI support windows: Major versions are supported for 12 months, minor versions for 6 months, and patch versions until a newer patch is released; AWS may end support earlier if core dependencies reach end-of-life.

Related Resources

• https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateClusterSoftware.html
• https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCluster.html
• https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateCluster.html
• https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-release-ami.html
• https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-ami-support-policy.html