What's New

Amazon SageMaker AI inference now supports G7 instances, powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs, delivering up to 4.6x AI inference performance compared to the previous-generation G6 instances. These instances are designed to address the cost and memory constraints that previously forced teams to over-provision compute or quantize models for production generative AI workloads. G7 instances are available immediately for real-time inference deployments via the SageMaker console, API, or SDK.

How It Works

• G7 instances are equipped with up to 8 NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs, each with 32 GB of GPU memory and 5th Generation Tensor Cores, providing high memory capacity and compute throughput for large model serving.
• The instances are powered by custom Intel Xeon 6 processors running at a sustained all-core turbo frequency of 3.9 GHz with simultaneous multi-threading disabled, maximizing per-core performance for CPU-to-GPU data movement in workloads like RAG inference and recommender systems.
• Up to 700 Gbps of EFA-enabled networking bandwidth (7x more than G6) enables low-latency, high-throughput communication, which is critical for multi-GPU inference and distributed model serving.
• Up to 7.6 TB of local NVMe SSD storage allows large model weights and datasets to reside close to compute, reducing data transfer overhead and improving inference throughput.
• Deployment is straightforward: users specify G7 instance types (ranging from ml.g7.xlarge to ml.g7.48xlarge) in their SageMaker endpoint configuration via the console, API, or SDK.
• The instances are built on the AWS Nitro System, which offloads networking and storage I/O to dedicated hardware, improving stability and enabling live firmware updates without downtime.

Why It's Important

• The 4.6x inference performance improvement over G6 instances allows teams to serve more requests per second at lower cost, directly reducing the per-inference cost for production generative AI applications.
• 32 GB of GPU memory per GPU eliminates the need to aggressively quantize models in the 7B–30B parameter range, preserving model quality and accuracy in production deployments.
• The 7x increase in EFA-enabled network bandwidth unlocks efficient multi-GPU and multi-node inference patterns, enabling teams to scale large model serving without networking becoming a bottleneck.
• Local NVMe SSD storage reduces cold-start latency for large models by keeping weights on-instance rather than requiring repeated fetches from remote storage like Amazon S3.
• Teams running multi-model endpoints benefit from the higher memory and bandwidth headroom, allowing more models to be co-located on a single instance and improving resource utilization.

How It's Different

• G7 instances use NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs, a newer GPU architecture compared to the Ada Lovelace-based GPUs in G6 instances, delivering 1.5x higher FP16 FLOPS and significantly improved tensor core throughput.
• Network bandwidth scales to 700 Gbps EFA on G7 versus 100 Gbps on G6, a 7x improvement that fundamentally changes the feasibility of distributed inference workloads on these instance families.
• G7 provides 32 GB of GPU memory per GPU, offering more headroom for large model weights compared to G6, reducing or eliminating the need for model quantization to fit within memory constraints.
• The custom Intel Xeon 6 CPU with SMT disabled is specifically tuned for high-bandwidth CPU-GPU data movement, making G7 better suited for RAG pipelines and recommender systems than general-purpose GPU instances.
• Ninth-generation NVENC and sixth-generation NVDEC engines on G7 support 4:2:2 encoding/decoding and deliver 1.6x more concurrent video streams than G6, making G7 a stronger choice for video generation and streaming workloads.

When to Prefer It

• Use G7 instances when serving open-source or fine-tuned generative AI models in the 7B–30B parameter range (e.g., Llama, Mistral, Qwen) where full-precision or lightly quantized weights require more than 16–24 GB of GPU memory.
• Prefer G7 for image and video generation workloads (e.g., Stable Diffusion, video diffusion models) that benefit from higher GPU memory bandwidth, advanced video encode/decode engines, and greater FP16 throughput.
• Choose G7 for multi-model inference endpoints where multiple models need to be loaded simultaneously, as the larger per-GPU memory pool allows more models to be co-resident without eviction.
• G7 is the right choice for RAG-based inference pipelines where frequent CPU-to-GPU data transfers are required, given the Xeon 6 processor's optimization for high-bandwidth data movement.
• Select G7 when cost-per-inference is a primary concern and current G6 deployments require over-provisioning multiple instances to meet throughput SLAs that a single G7 instance could handle.
• Use G7 for latency-sensitive production endpoints where the combination of higher GPU throughput and faster local NVMe model loading reduces both steady-state and cold-start latency.

Availability

• G7 instances for SageMaker AI inference are generally available (GA) as of July 22, 2026.
• Currently available in three AWS regions: US East (N. Virginia), US East (Ohio), and US West (Oregon).
• Supported instance sizes range from ml.g7.xlarge through ml.g7.48xlarge, providing flexibility to right-size deployments from single-GPU to 8-GPU configurations.
• Pricing follows the standard SageMaker AI on-demand model with no upfront commitments; Amazon SageMaker Savings Plans are also applicable for committed usage discounts.
• Specific per-hour pricing is available on the SageMaker AI pricing page (https://aws.amazon.com/sagemaker/ai/pricing/).
• No mention of support for SageMaker batch transform or asynchronous inference in the announcement; the feature is confirmed for real-time inference endpoints.

Related Resources

• https://aws.amazon.com/ec2/instance-types/g7/
• https://aws.amazon.com/sagemaker/ai/pricing/?refid=ft_sagemaker