Amazon SageMaker HyperPod now supports custom AMIs (Amazon Machine Images) for Slurm clusters
Bake security agents and custom drivers into HyperPod Slurm nodes at the image level — faster startups, zero config drift, enterprise compliance met.
View original announcement →Visual Summary
graph TD
A{{HyperPod Custom AMIs}}:::announced
B(SageMaker HyperPod):::compute
C([HyperPod Base AMIs]):::feature
D([EC2 Image Builder]):::feature
E((Security Teams)):::external
F((AI/ML Teams)):::external
G([Slurm Cluster Nodes]):::feature
H([CreateCluster API]):::feature
I(Distributed Training):::compute
E ==>|"builds hardened images"| A
A -->|"extends"| C
D -->|"customizes"| A
F -->|"specifies AMI"| H
H ==>|"deploys"| B
B -->|"boots nodes from"| A
B -->|"provisions"| G
G -->|"runs"| I
C -.->|"preserves compatibility"| I
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon SageMaker HyperPod now supports custom Amazon Machine Images (AMIs) for Slurm-orchestrated clusters, allowing customers to bake security agents, compliance tools, proprietary libraries, and specialized drivers directly into cluster node images. Built on top of HyperPod's performance-optimized public base AMIs, these custom images replace complex, error-prone lifecycle configuration scripts with pre-hardened, consistent environments. The feature is accessible via the CreateCluster, UpdateCluster, and UpdateClusterSoftware APIs, covering new cluster creation, instance group additions, and in-place patching of existing clusters.
How It Works
- Base AMI requirement: Custom AMIs must be derived from HyperPod's publicly published, performance-tuned base AMIs to preserve compatibility with distributed training libraries (e.g., NCCL, EFA drivers) and HyperPod's cluster management plane.
- Image customization layer: Customers extend the base AMI by installing security agents, compliance tooling, proprietary ML libraries, custom CUDA/driver versions, or any other runtime dependency using standard AMI-building toolchains (e.g., EC2 Image Builder, Packer).
- API integration: The custom AMI ID is specified at the instance-group level when calling
CreateCluster(new clusters),UpdateCluster(adding instance groups), orUpdateClusterSoftware(patching existing nodes), giving fine-grained control per node group. - Compatibility validation: The HyperPod APIs automatically validate image compatibility and handle IAM permission checks before the AMI is applied to cluster nodes, preventing misconfigured images from reaching production.
- Startup behavior: Because all software is pre-baked into the AMI, nodes boot directly into a fully configured state, eliminating the post-launch bootstrapping phase previously handled by lifecycle scripts.
- Backwards compatibility: Existing clusters using lifecycle scripts continue to function; the custom AMI path is additive and does not break current workflows.
Why It's Important
- Enterprise security unblocked: Organizations with strict security postures (FedRAMP, HIPAA, SOC 2, internal hardening standards) can now embed approved security agents and policies at the image level rather than fighting against runtime script limitations, removing a major adoption barrier.
- Faster time-to-training: Pre-baked images eliminate lengthy bootstrapping sequences on cluster startup, reducing the time between cluster launch and the first training job, which is critical for large GPU clusters where idle time is expensive.
- Consistency and reliability: All nodes in an instance group boot from an identical image, eliminating configuration drift and the class of bugs caused by partial or failed lifecycle script execution across nodes.
- Separation of concerns: Security teams own and publish hardened AMIs; AI/ML platform teams consume pre-approved images without needing to understand or modify security tooling, improving organizational agility and audit trails.
- Reduced operational complexity: Replacing multi-step lifecycle scripts with a single AMI reference simplifies cluster definitions, makes rollbacks straightforward (swap AMI version), and improves reproducibility across dev/staging/production environments.
How It's Different
- vs. lifecycle configuration scripts: Scripts run post-launch on every node boot, are prone to network failures, version drift, and slow startup; custom AMIs are immutable, pre-validated, and execute no bootstrapping logic at launch time.
- vs. generic EC2 custom AMIs: HyperPod custom AMIs must inherit from HyperPod's official base images, ensuring that EFA networking, NCCL optimizations, Slurm integration, and the HyperPod health monitoring agent remain intact — something a fully custom EC2 AMI cannot guarantee.
- vs. container-only customization (e.g., Docker images): Container images customize the application layer but cannot modify kernel modules, low-level drivers, or host-level security daemons; custom AMIs operate at the OS/host level, covering the full software stack.
- vs. AWS ParallelCluster custom AMIs: HyperPod's managed control plane handles AMI validation, permission management, and cluster resilience automatically, whereas ParallelCluster requires more manual orchestration of these concerns.
- Integrated API validation: Unlike ad-hoc AMI substitution, HyperPod's APIs actively validate AMI lineage and permissions before deployment, reducing the risk of incompatible images silently degrading cluster performance.
When to Prefer It
- Regulated industries: Use custom AMIs when your organization must demonstrate that every compute node runs a certified, audited software baseline (e.g., financial services, healthcare, government) and runtime script installation cannot satisfy auditors.
- Large-scale GPU clusters: Prefer custom AMIs when cluster startup latency directly impacts cost or SLA, particularly for clusters with hundreds of p4d/p5 instances where bootstrapping delays compound significantly.
- Proprietary or licensed software: Use custom AMIs when you need to embed licensed ML frameworks, internal libraries, or vendor-supplied drivers that cannot be distributed via public package managers or installed reliably at runtime.
- Multi-team platform environments: Ideal when a central platform or security team needs to publish a catalog of approved images that multiple AI/ML teams consume, enforcing standardization without restricting team autonomy.
- Frequent cluster cycling: When clusters are frequently created and destroyed (e.g., spot-based training bursts), custom AMIs maximize the benefit of fast, consistent node initialization over repeated lifecycle script execution.
- Compliance-driven patching workflows: Use
UpdateClusterSoftwarewith a new AMI version to implement controlled, auditable OS/security patch rollouts across existing clusters without redeploying from scratch.
Availability
- GA status: Generally available as of July 13, 2026; this is not a preview feature.
- Regional availability: Available in all AWS Regions where Amazon SageMaker HyperPod is currently supported; no region-specific restrictions were announced.
- Orchestrator scope: Custom AMI support applies to Slurm-orchestrated HyperPod clusters only; availability for Amazon EKS-orchestrated HyperPod clusters was not announced in this release.
- Base AMI requirement (hard constraint): Custom AMIs must be built from HyperPod's official public base AMIs; AMIs built from arbitrary third-party or self-generated base images are not supported and will fail API validation.
- Pricing: No additional charge for the custom AMI feature itself; standard SageMaker HyperPod instance and storage pricing applies, along with any EC2 costs incurred during AMI build processes.
- Documentation: Full guidance on building a custom AMI and integrating it with cluster APIs is available in the Amazon SageMaker HyperPod User Guide at the link provided in the announcement.
