What's New

Amazon SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), an inference optimization that physically separates the compute-intensive prefill phase and the memory-bandwidth-intensive decode phase of LLM inference onto dedicated GPU pools. KV cache is transferred between these pools over Elastic Fabric Adapter (EFA) using GPU-Direct RDMA, eliminating resource contention between the two phases. This capability is enabled via a simple pdSpec addition to the existing InferenceEndpointConfig custom resource and is composable with HyperPod's existing KV cache offloading and intelligent routing features.

How It Works

• Phase separation: Prefill (processing the input prompt, compute-bound) runs on one dedicated GPU pool, while decode (generating output tokens one at a time, memory-bandwidth-bound) runs on a separate GPU pool, so neither phase starves the other of its primary resource.
• KV cache transfer: After prefill completes, the resulting key-value cache is transferred from the prefill pool to the decode pool over EFA using GPU-Direct RDMA, enabling low-latency, high-bandwidth data movement without CPU involvement.
• Intelligent routing: An embedded router inspects each incoming request and automatically routes long-context requests through the disaggregated prefill→decode path, while sending short prompts directly to the decode pool to avoid unnecessary transfer overhead.
• Orchestration via Inference Operator: The HyperPod Inference Operator handles all orchestration automatically — provisioning the router, wiring prefill and decode pods together via LMCache and NIXL, and integrating with HyperPod observability.
• Configuration: DPD is enabled by adding a pdSpec section to the existing InferenceEndpointConfig custom resource, requiring no new resource types or major workflow changes.
• Independent scaling: Prefill and decode pools can be scaled independently, allowing operators to match capacity to the actual input/output token distribution of their workload.

Why It's Important

• Eliminates head-of-line blocking: In colocated deployments, a single long-context request monopolizes the GPU during prefill, stalling token generation for all concurrent requests. DPD removes this interference entirely.
• Predictable per-token latency: By isolating decode onto dedicated GPUs, inter-token latency (ITL) becomes stable under sustained concurrency, which is critical for real-time user-facing applications like chat assistants.
• Higher goodput at strict SLOs: Teams can serve more requests within latency SLOs without over-provisioning, directly reducing infrastructure cost per request.
• Independent capacity management: Prefill and decode have fundamentally different resource profiles; disaggregation allows right-sizing each pool separately rather than forcing a single compromise configuration.
• Reduces over-provisioning: Previously, customers had to over-provision one phase to protect the other from interference; DPD eliminates this inefficiency.
• Composability: DPD integrates with existing HyperPod features (KV cache offloading, intelligent routing), so teams can layer optimizations without architectural rewrites.

How It's Different

• vs. colocated inference (standard vLLM/TGI deployments): Traditional deployments run prefill and decode on the same GPU, causing mutual interference under mixed traffic; DPD physically separates them, eliminating contention at the hardware level.
• vs. manual multi-node setups: Achieving similar separation without DPD would require custom orchestration, manual pod wiring, and bespoke routing logic; HyperPod's Inference Operator automates all of this.
• vs. simply scaling up replicas: Adding more colocated replicas reduces per-replica load but does not eliminate the within-replica interference between a long prefill and concurrent decodes; DPD addresses the root cause.
• Smart routing differentiator: The intelligent router avoids applying disaggregation to short prompts where the KV transfer overhead would outweigh the benefit, making DPD self-optimizing across mixed traffic patterns.
• GPU-Direct RDMA transport: KV cache transfer bypasses the CPU entirely using EFA + GPU-Direct RDMA, keeping transfer latency low enough that disaggregation is practical even for latency-sensitive workloads.
• Composable architecture: Unlike standalone disaggregation research implementations, HyperPod DPD is composable with KV cache offloading and existing routing, enabling multi-layered optimization within a single managed platform.

When to Prefer It

• Large dense models (70B+ parameters): Models like Llama 3.3 70B generate large KV caches during prefill, making the interference problem severe and the benefit of disaggregation proportionally high.
• Long input contexts (4,000+ tokens): Longer prompts cause longer prefill stalls for concurrent decode requests; ITL improvement scales with input length, so DPD delivers the most value here.
• Sustained concurrency (2+ RPS): Without multiple concurrent requests competing for the same GPU, there is nothing to disaggregate; DPD is most impactful when the GPU is consistently shared across requests.
• Moderate-to-long outputs (256+ tokens): More output tokens means more cumulative benefit from stable per-token latency, making DPD worthwhile for generative tasks with substantial output length.
• Strict latency SLO environments: Applications like chat assistants, agentic pipelines, and RAG systems where tail latency directly impacts user experience or downstream pipeline reliability benefit most.
• Mixed traffic workloads: When the request distribution includes both short and long prompts, the intelligent router ensures DPD is applied selectively, making it safe to enable without penalizing short-prompt traffic.
• Avoid for small models or low concurrency: If the workload uses small models, short inputs, or low request rates, a standard colocated deployment is simpler and performs comparably without the operational overhead of managing separate pools.

Availability

• GA status: Generally available as of July 6, 2026.
• Orchestrator requirement: Requires SageMaker HyperPod clusters using the EKS orchestrator; not available on Slurm-based HyperPod clusters.
• Instance type requirement: Only supported on EFA-capable instance types (e.g., p4d, p4de, p5, trn1 families) due to the GPU-Direct RDMA KV cache transfer mechanism.
• Regional availability: Available in all AWS Regions where Amazon SageMaker HyperPod is available.
• Pricing: No separate charge for the DPD feature itself; customers pay standard SageMaker HyperPod and EC2 instance pricing for the prefill and decode GPU pools they provision.
• Known limitations: Documentation notes known limitations (detailed in the developer guide); users should review these before production deployment, particularly around supported model architectures and maximum context lengths.
• Prerequisites: Requires AWS CLI, kubectl access to the HyperPod EKS cluster, and (for gated models) a Hugging Face token with read access to the model checkpoint.

Related Resources

• https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-dpd.html