What's New

Amazon SageMaker HyperPod now supports partition-level network topology configuration for Slurm-orchestrated clusters, allowing a single cluster to simultaneously run different topology types (block or tree) across different partitions based on the instance types present. This means workloads on UltraServer instances (e.g., ml.p6e-gb200.36xlarge) automatically use block topology, while hierarchical-interconnect instances (e.g., ml.p5.48xlarge) use tree topology — all within the same cluster. The feature is enabled by default on Slurm 25.11+ with no manual configuration required, and HyperPod dynamically maintains topology accuracy through scale-up, scale-down, and node replacement events.

How It Works

• Partition-level topology assignment: HyperPod inspects the instance types assigned to each Slurm partition and automatically selects the appropriate topology plugin — block topology for UltraServer instances, tree topology for hierarchical-interconnect instances, and no topology constraint for instance types without topology metadata.
• Block topology for UltraServer (ml.p6e-gb200.36xlarge): Up to 18 p6e-gb200.36xlarge instances within a single UltraServer are interconnected via NVLink switches, enabling GPU-to-GPU data transfer across all nodes without traversing traditional network interfaces. Block topology ensures Slurm schedules jobs across nodes that share this NVLink fabric.
• Tree topology for hierarchical instances (ml.p5.48xlarge, ml.p5e.48xlarge, ml.p5en.48xlarge): These instances use a multi-level network switch hierarchy. Tree topology maps this hierarchy so Slurm places jobs on nodes that share the fewest network hops, minimizing inter-node communication latency.
• Automatic lifecycle management: As the cluster scales up or down, or nodes are replaced, HyperPod recalculates and updates the topology configuration for each partition automatically, ensuring scheduling decisions always reflect the live cluster state.
• NCCL and GPU communication optimization: By aligning job placement with physical interconnect topology, collective operations (AllReduce, AllGather, etc.) used in distributed training traverse the most efficient communication paths, reducing synchronization overhead.
• Requires Slurm 25.11+: The feature leverages topology plugins introduced or stabilized in Slurm 25.11 and is activated by default upon cluster creation or upgrade — no user-side topology configuration files are needed.

Why It's Important

• Direct training throughput improvement: Topology-aware scheduling eliminates the performance penalty of placing communicating GPUs on nodes that are physically far apart in the network hierarchy, which is one of the most common sources of distributed training slowdowns at scale.
• Heterogeneous cluster support without compromise: Organizations running mixed workloads — e.g., large-scale pre-training on p6e UltraServers alongside fine-tuning on p5 instances — can now do so within a single HyperPod cluster without sacrificing optimal scheduling for either workload type.
• Reduced operational burden: Previously, operators had to manually manage topology configuration files or maintain separate clusters per instance family. Automatic partition-level topology removes this toil entirely.
• NCCL efficiency at scale: For large model training jobs with thousands of GPUs, even small improvements in collective communication efficiency compound significantly, translating to measurable reductions in time-to-train and compute cost.
• Resilience through dynamic updates: Because topology is recalculated on node replacement and scaling events, transient hardware failures or elastic scaling no longer risk leaving the cluster in a stale or suboptimal topology state.

How It's Different

• Partition-level granularity vs. cluster-level: Previously, topology configuration in HyperPod (and Slurm generally) was applied uniformly across a cluster. This release introduces per-partition topology, enabling a single cluster to host instance families with fundamentally different interconnect architectures simultaneously.
• Automatic topology detection vs. manual configuration: Standard Slurm topology setup requires administrators to author and maintain topology.conf files. HyperPod derives topology automatically from EC2 instance metadata, eliminating manual file management and the risk of configuration drift.
• UltraServer block topology support: Block topology is a distinct Slurm plugin suited to flat, fully-connected fabrics like NVLink-switched UltraServers. This is architecturally different from tree topology and was not previously supported at the partition level in HyperPod.
• Dynamic lifecycle awareness: Unlike static topology configurations, HyperPod's implementation updates topology in real time as the cluster changes, which is critical for elastic training workloads that scale nodes up and down frequently.
• Zero-configuration enablement: The feature is on by default — users do not need to opt in, set flags, or modify job submission scripts to benefit from topology-aware scheduling.

When to Prefer It

• Large-scale distributed pre-training on UltraServer instances: When running foundation model training jobs across multiple ml.p6e-gb200.36xlarge instances, block topology ensures all GPUs within an UltraServer's NVLink domain are treated as a scheduling unit, maximizing NVLink bandwidth utilization.
• Mixed-instance HyperPod clusters: When a single cluster serves multiple teams or workload types using different GPU instance families (e.g., p5 for fine-tuning, p6e for pre-training), partition-level topology ensures each partition is optimally scheduled without requiring separate clusters.
• NCCL-heavy workloads with frequent AllReduce operations: Training jobs using data parallelism or tensor parallelism with frequent gradient synchronization benefit most from topology-aware placement, as it minimizes the number of network hops between communicating ranks.
• Elastic training pipelines with frequent scaling: When clusters scale up or down dynamically (e.g., spot-based or demand-driven scaling), automatic topology recalculation ensures newly added nodes are immediately scheduled with correct topology awareness.
• Clusters upgrading to Slurm 25.11+: Any existing HyperPod Slurm cluster being upgraded to Slurm 25.11 with supported GPU instance types should leverage this feature as a default performance improvement with no additional effort.

Availability

• General Availability: This feature is generally available as of July 17, 2026.
• Supported regions: Available in all AWS Regions where Amazon SageMaker HyperPod is supported.
• Slurm version requirement: Requires Slurm 25.11 or later; clusters on earlier Slurm versions must be upgraded to benefit.
• Supported instance types: Block topology is supported for ml.p6e-gb200.36xlarge (UltraServer); tree topology is supported for ml.p5.48xlarge, ml.p5e.48xlarge, and ml.p5en.48xlarge. Instance types without topology metadata remain schedulable without topology constraints.
• Pricing: No additional charge for the topology-aware scheduling feature itself; standard SageMaker HyperPod and EC2 instance pricing applies.
• Configuration: Enabled by default; no user configuration required. Applies to both newly created clusters and existing clusters updated to Slurm 25.11+.
• EKS note: Topology-aware scheduling also works with EKS-orchestrated HyperPod clusters (via Kubernetes topology labels), but this announcement specifically covers the Slurm partition-level enhancement.

Related Resources

• https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-topology.html