Amazon SageMaker HyperPod now supports AMI-based node lifecycle configuration for Slurm clusters using continuous provisioning
Skip the S3 scripts entirely — HyperPod now auto-configures Slurm nodes from an AMI as capacity scales in, cutting setup time and ops overhead.
View original announcement →Visual Summary
graph TD
A{{AMI-based Lifecycle Config\nfor Continuous Provisioning}}:::announced
B(SageMaker HyperPod):::compute
C([Slurm Cluster]):::feature
D(Amazon S3):::storage
E([Extension Scripts]):::feature
F([Pre-baked AMI]):::feature
G((ML Engineers)):::external
H([Continuous Provisioning]):::feature
I([Docker/Enroot/Pyxis]):::feature
G ==>|"creates cluster"| B
B ==>|"provisions nodes"| A
A ==>|"configures from"| F
F -->|"includes"| I
A -->|"joins nodes"| C
H -->|"adds capacity"| A
A -.->|"optional customization"| E
E -.->|"stored in"| D
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon SageMaker HyperPod now extends AMI-based node lifecycle configuration to Slurm clusters that use continuous provisioning, the mode that adds nodes to a cluster incrementally as capacity becomes available. Previously, AMI-based configuration was not available for this provisioning mode, requiring users to author, upload, and manage lifecycle scripts in Amazon S3. With this launch, teams can spin up production-ready Slurm clusters without writing or maintaining any lifecycle scripts, while still retaining the option to layer on extension scripts or fully custom scripts when needed.
How It Works
- AMI-based provisioning baseline: Each node is bootstrapped directly from a pre-baked AMI that includes production-ready software (Docker, Enroot, Pyxis) and configurations (Slurm accounting, SSH key generation, log rotation), eliminating the need to download or execute external setup scripts at launch time.
- Continuous provisioning integration: As capacity becomes available and new nodes are added to the cluster, each node is individually configured from the AMI on join, so the cluster grows without requiring any manual intervention or script management per node.
- Enabling AMI-based config via API: Omit the
LifeCycleConfigblock entirely from the instance group configuration in theCreateClusterAPI call; HyperPod automatically applies AMI-based configuration. - Enabling via console: In the SageMaker AI console under Custom setup, select "None" under Lifecycle scripts to activate AMI-based configuration.
- Optional extension scripts: For additional customization on top of the AMI baseline, specify the
OnInitCompleteparameter andSourceS3Uriin theLifeCycleConfigblock (API) or provide an S3 URI in the "Extension script file in S3" field (console); only the delta customization script needs to be managed. - Full custom scripts still supported: Teams that require complete control over node provisioning can continue supplying fully custom lifecycle scripts, which remain a first-class option.
- Slurm topology is independent: Node lifecycle configuration (AMI-based, extension, or custom) is a separate axis from Slurm topology configuration (API-driven
SlurmConfigor legacyprovisioning_parameters.json), and the two can be combined freely.
Why It's Important
- Dramatically lowers the barrier to entry: New users no longer need to understand, write, test, or maintain lifecycle shell scripts just to get a functional Slurm cluster, reducing onboarding time from hours to minutes.
- Faster node readiness in continuous provisioning: Because nodes are configured from the AMI rather than running scripts at boot, each newly added node becomes schedulable sooner, improving cluster elasticity and reducing job queue wait times.
- Reduced operational burden: Eliminating S3-hosted lifecycle scripts removes an entire class of operational artifacts (versioning, IAM permissions, script debugging) that teams previously had to manage.
- Production-ready out of the box: The AMI includes all components required for real AI/ML training workloads (container runtimes, Slurm integrations, logging), so clusters are immediately usable without additional hardening.
- Flexibility is preserved: The three-tier model (AMI-only → AMI + extension script → fully custom) means teams can adopt the simplest option that meets their needs and graduate to more control only when required.
- Consistent node state: AMI-baked configuration ensures every node in the cluster starts from an identical, tested baseline, reducing configuration drift that can occur with script-based approaches.
How It's Different
- No S3 script management required: Unlike the traditional lifecycle script approach, AMI-based configuration requires zero S3 artifacts for a fully functional cluster, whereas custom scripts require uploading and maintaining scripts in S3 before cluster creation.
- Faster node join time: AMI-based nodes skip the script download and execution phase at boot, making them available to the Slurm scheduler faster than nodes that run lifecycle scripts on first boot.
- Continuous provisioning now on par with static provisioning: Previously, AMI-based configuration was only available for statically provisioned clusters; this launch closes the gap so both provisioning modes offer the same simplified setup experience.
- Layered customization model: Rather than an all-or-nothing choice, HyperPod now offers a graduated customization ladder (AMI → AMI + extension → full custom), whereas previously users had to commit to full script management even for minor customizations.
- Immutable baseline vs. mutable scripts: The AMI baseline is versioned and tested by AWS, providing a more reliable starting point than user-managed scripts that can diverge across cluster versions or team members.
When to Prefer It
- New teams getting started with HyperPod: If your team has no existing lifecycle scripts and wants the fastest path to a working Slurm cluster, AMI-based configuration is the recommended default.
- Elastic workloads with continuous provisioning: When your training jobs require the cluster to scale out dynamically as GPU capacity becomes available, AMI-based configuration ensures new nodes join and become schedulable with minimal delay.
- Standard AI/ML training environments: If your workloads rely on common frameworks and container runtimes (Docker, Enroot, Pyxis) and do not require exotic system-level customizations, the AMI baseline covers all prerequisites.
- Teams wanting minimal operational overhead: Organizations that want to reduce the number of AWS artifacts (S3 buckets, IAM policies for script access, script versioning) they manage should prefer AMI-based configuration.
- AMI + extension script for moderate customization: If you need to install a specific library, mount an additional filesystem, or set environment variables beyond the baseline, use the extension script option rather than writing a full lifecycle script from scratch.
- Stick with custom scripts when full control is required: If your environment mandates custom kernel modules, non-standard Slurm plugins, compliance-driven hardening steps, or complex multi-step provisioning logic, continue using fully custom lifecycle scripts.
Availability
- GA status: Generally available; this is a production launch, not a preview.
- Supported regions: Available in all AWS Regions where Amazon SageMaker HyperPod is currently offered.
- Provisioning mode requirement: Applies specifically to Slurm clusters using continuous provisioning; AMI-based configuration was already available for other provisioning modes.
- Pricing: No additional charge for AMI-based configuration; standard SageMaker HyperPod and underlying EC2 instance pricing applies.
- Backward compatibility: Existing clusters using custom lifecycle scripts are unaffected; custom scripts remain fully supported and no migration is required.
- Console and API support: Feature is accessible via both the SageMaker AI console (Quick setup and Custom setup flows) and the AWS CLI / SageMaker API.
- Documentation: Getting started guides are available for both the AWS CLI and the SageMaker AI console in the SageMaker Developer Guide.
