What's New

Amazon SageMaker HyperPod now supports a MinInstanceCount (MinCount) parameter for Slurm-orchestrated clusters running in continuous provisioning mode. This feature lets teams specify the minimum number of instances that must be successfully provisioned before an instance group transitions to InService status and becomes available for job scheduling. It bridges the gap between the speed benefits of continuous provisioning and the fixed-topology requirements of distributed training frameworks like PyTorch FSDP, Megatron-LM, and NVIDIA NeMo.

How It Works

• MinInstanceCount parameter: Specified in the CreateCluster or UpdateCluster API request at the instance group level, it defines the minimum number of nodes that must be provisioned before the group is considered ready.
• Status gating: The instance group remains in Creating or Updating status until the MinCount threshold is met; only then does it transition to InService, making nodes visible to the Slurm scheduler.
• Continued provisioning beyond MinCount: After the threshold is satisfied and the group enters InService, HyperPod continues launching additional instances asynchronously until the full target count (InstanceCount) is reached.
• Automatic rollback on failure: If the MinCount threshold cannot be satisfied within a 3-hour window, the system automatically rolls back the instance group to its last known good state, preventing a partially degraded cluster from persisting.
• Built on continuous provisioning: This feature requires NodeProvisioningMode set to Continuous on the cluster, which uses priority-based provisioning (controller node first, then login and worker nodes in parallel) and retries failed node launches asynchronously.

Why It's Important

• Enables fixed-topology distributed training: Frameworks like PyTorch FSDP, Megatron-LM, and NVIDIA NeMo require a precise number of participating nodes; starting with fewer can cause jobs to fail or produce incorrect results, and MinCount prevents premature job scheduling in such scenarios.
• Reduces wasted compute spend: Teams can now guarantee a baseline GPU count is available before committing to an expensive training run, avoiding the cost of launching a job that immediately fails due to insufficient capacity.
• Supports SLA enforcement: Organizations with contractual or operational SLAs on training throughput can use MinCount to ensure a minimum resource floor is met before workloads begin.
• Eliminates the all-or-nothing dilemma: Previously, teams had to choose between waiting for full cluster provisioning (slow) or risking an under-resourced job start; MinCount provides a configurable middle ground.
• Automated failure recovery: The 3-hour rollback mechanism removes the need for manual intervention when capacity cannot be satisfied, keeping clusters in a known-good state without operator involvement.

How It's Different

• vs. standard continuous provisioning (no MinCount): Without MinCount, an instance group transitions to InService as soon as the minimum controller/login nodes are ready, potentially exposing worker slots to the Slurm scheduler before enough GPU nodes exist for a distributed job; MinCount adds an explicit, user-defined readiness gate.
• vs. traditional all-or-nothing provisioning: The legacy model blocked the entire cluster creation if any instance group fell short of its full target count; MinCount allows partial provisioning to succeed while still enforcing a meaningful capacity floor, combining resilience with control.
• vs. manual polling/scripting: Previously, teams would need external scripts to poll cluster status and hold job submissions until enough nodes were available; MinCount makes this a first-class, API-native capability with automatic state management.
• Scoped to instance groups: MinCount is set per instance group, not per cluster, giving fine-grained control over which node pools require a capacity guarantee and which can tolerate partial availability.

When to Prefer It

• Large-scale FSDP or tensor-parallel training runs: When your training script is hard-coded to a specific world size (e.g., 64 or 128 GPUs), use MinCount to ensure that exact node count is available before the Slurm scheduler can dispatch the job.
• Megatron-LM or NeMo pre-training jobs: These frameworks partition model layers across a fixed pipeline and tensor-parallel topology; an insufficient node count at job start causes immediate failure, making MinCount essential.
• Cost-gated training commitments: When a training run only makes economic sense above a certain GPU count (e.g., to achieve target MFU), set MinCount to that threshold to avoid committing reserved instance hours to an inefficient run.
• SLA-bound production training pipelines: When downstream teams or customers depend on model delivery timelines, MinCount ensures training doesn't begin until the infrastructure can realistically meet throughput targets.
• Capacity-constrained regions with spot-like availability: In regions where large GPU instance types are intermittently available, MinCount combined with the 3-hour retry window maximizes the chance of reaching a viable cluster size before falling back gracefully.
• Multi-instance-group clusters with heterogeneous requirements: Use MinCount selectively on GPU worker groups while leaving CPU or utility groups ungated, so the cluster enters service as soon as the critical compute tier is ready.

Availability

• General Availability: This feature is generally available as of May 27, 2026; no preview or beta designation is indicated.
• Prerequisite: Requires NodeProvisioningMode set to Continuous on the HyperPod cluster; MinCount is not available for clusters using the default (non-continuous) provisioning mode.
• Orchestrator scope: Supported only for HyperPod clusters using the Slurm orchestrator; availability for EKS-based HyperPod clusters is not mentioned in this announcement.
• Regional availability: Available in all AWS Regions where Amazon SageMaker HyperPod is currently supported; no region-specific restrictions are noted.
• Pricing: No additional charge is associated with the MinCount feature itself; standard SageMaker HyperPod and EC2 instance pricing applies.
• Timeout limitation: If the MinCount threshold is not met within 3 hours of provisioning initiation, the system automatically rolls back the instance group; there is no configurable timeout window at this time.
• API surface: Configured via MinInstanceCount in the CreateCluster or UpdateCluster API requests; also accessible through the AWS CLI and SageMaker AI console.

Related Resources

• https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-scaling-slurm.html#sagemaker-hyperpod-scaling-slurm-mincount