Amazon SageMaker HyperPod now supports EFA-only network interfaces
Scale HyperPod clusters further by eliminating per-interface IP address consumption — deploy more nodes without VPC subnet exhaustion.
View original announcement →Visual Summary
graph TD
A{{EFA-Only Interfaces}}:::announced
B(SageMaker HyperPod):::compute
C([IP Exhaustion Prevention]):::feature
D([High-Throughput Comms]):::feature
E(VPC Subnet):::storage
F((ML Engineers)):::external
G([Mixed Interface Config]):::feature
H(CreateCluster API):::compute
I([Distributed Training]):::feature
F ==>|"configures"| H
H ==>|"specifies efa-only"| A
A ==>|"provisions on"| B
A -->|"eliminates"| C
C -->|"preserves capacity"| E
A -->|"enables"| D
A -.->|"supports"| G
B -->|"runs"| I
D -->|"accelerates"| I
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon SageMaker HyperPod now supports EFA-only network interfaces, allowing cluster instance groups to be configured with dedicated Elastic Fabric Adapter (EFA) devices that operate without an accompanying Elastic Network Adapter (ENA) for IP networking. This new efa-only interface type eliminates the mandatory IP address consumption that previously occurred with every EFA interface, directly addressing VPC subnet exhaustion at scale. The feature is available globally across all AWS Regions where SageMaker HyperPod is supported and is configurable via the CreateCluster/UpdateCluster APIs.
How It Works
- Standard
efainterface type: When using the existingefainterface type, each network interface attaches both an EFA device (for high-performance inter-node communication) and an ENA device (for IP networking), consuming one IP address per interface from the VPC subnet — regardless of whether IP connectivity is needed on that interface. - New
efa-onlyinterface type: SettingInterfaceTypetoefa-onlyin theClusterNetworkInterfaceconfiguration provisions the network interface exclusively for EFA traffic, with no ENA device attached and no IP address allocated from the subnet. - Mixed interface configuration: A node can combine both interface types — using
efaon a subset of interfaces that require IP networking andefa-onlyon the remaining interfaces dedicated purely to high-throughput inter-node communication. - API integration: The
efa-onlyvalue is specified in theClusterNetworkInterfaceobject (field:InterfaceType) when callingCreateClusterorUpdateCluster, and is also supported on cluster updates to existing instance groups. - EFA capabilities preserved: EFA-only interfaces retain full EFA functionality — including OS-bypass, low-latency RDMA-like semantics, and high-throughput collective communication — the only omission is the IP networking layer provided by ENA.
Why It's Important
- Eliminates IP exhaustion at scale: Large GPU clusters (e.g., instances like
p5.48xlargewith up to 32 EFA interfaces per node) previously consumed enormous numbers of IP addresses;efa-onlyremoves this constraint, enabling far larger clusters within existing VPC subnets. - Unlocks denser cluster deployments: Organizations no longer need to re-architect their VPC CIDR blocks or split clusters across multiple subnets solely to accommodate IP address demand from EFA interfaces, reducing operational complexity.
- Improves training performance economics: By maximizing the number of EFA interfaces dedicated to inter-node communication without networking overhead trade-offs, teams can fully exploit available bandwidth for collective operations (AllReduce, AllGather) in distributed training.
- Reduces infrastructure re-planning: Teams running at the boundary of subnet capacity can now scale up existing HyperPod clusters without requesting VPC CIDR expansions or subnet redesigns, accelerating time-to-training.
- Aligns with large-scale LLM/foundation model trends: As model sizes and cluster sizes continue to grow, IP address management has become a real operational bottleneck; this feature directly addresses a known pain point for frontier model training teams.
How It's Different
- No ENA attachment: Unlike the standard
efatype,efa-onlydoes not attach an ENA device to the interface, meaning no IP address is allocated from the subnet for that interface — a fundamental architectural difference. - IP address footprint: A node with 32 EFA interfaces using the standard
efatype consumes 32 IP addresses; withefa-onlyon 31 of those interfaces (keeping oneefafor management), only 1 IP address is consumed — a ~97% reduction in IP usage per node. - Purpose-specific design: The
efa-onlytype is purpose-built for pure inter-node communication workloads, whereasefais a dual-purpose interface that serves both HPC communication and standard IP networking. - Complementary, not a replacement:
efa-onlydoes not replaceefa; at least oneefa(or standard ENA) interface is still required per node for control-plane IP connectivity, making the two types complementary in a mixed configuration. - Subnet scalability: With
efa-only, the number of nodes deployable in a single subnet is limited primarily by instance availability and subnet size relative to management interfaces, not by the multiplicative IP consumption of all EFA interfaces.
When to Prefer It
- Large-scale distributed training: Use
efa-onlywhen running multi-node training jobs (hundreds to thousands of nodes) where inter-node collective communication bandwidth is the primary bottleneck and most EFA interfaces are used exclusively for MPI/NCCL traffic. - IP-constrained VPC environments: Choose
efa-onlywhen your VPC subnets have limited CIDR ranges and you are approaching or have hit IP address exhaustion due to the number of EFA interfaces per instance. - Instances with many EFA interfaces: Particularly beneficial for high-EFA-count instances (e.g.,
p5.48xlarge,p5e.48xlarge,trn2.48xlarge) where the IP overhead per node is highest and the benefit of eliminating ENA devices is most pronounced. - Homogeneous training clusters: Ideal when all nodes in an instance group perform the same role (pure compute/training) and do not require independent IP reachability on every network interface for application-level communication.
- Cost-sensitive subnet planning: Use when you want to avoid the cost and complexity of allocating larger VPC CIDRs or additional subnets purely to accommodate EFA interface IP requirements.
- Avoid
efa-onlywhen: Any interface needs to carry IP traffic (e.g., storage access, control-plane communication, or direct SSH to a specific interface) — in those cases, retain the standardefatype for those specific interfaces.
Availability
- General Availability (GA): This feature is generally available as of June 1, 2026 — no preview or waitlist required.
- Regional availability: Available in all AWS Regions where Amazon SageMaker HyperPod is currently supported; no region-specific exclusions were announced.
- Pricing: No additional charge for using
efa-onlyinterfaces; standard SageMaker HyperPod and EC2 instance pricing applies. - API requirement: Configured via the
InterfaceTypefield in theClusterNetworkInterfaceobject using theCreateClusterorUpdateClusterSageMaker API calls. - Limitation — no IP networking: EFA-only interfaces cannot carry IP traffic; at least one non-
efa-onlyinterface per node is required for management and control-plane connectivity. - Instance compatibility: Only applicable to EFA-capable instance types supported by SageMaker HyperPod; standard CPU or non-EFA instances are unaffected by this configuration option.
