What's New

Amazon SageMaker HyperPod now includes native data capture for inference workloads, automatically recording request and response payloads from production endpoints to Amazon S3. The feature eliminates the need for custom logging pipelines by providing three configurable capture tiers — at the SageMaker endpoint, the Application Load Balancer, and the model pod — each targeting a different point in the inference request flow. Captured data integrates directly with SageMaker Model Monitor, fine-tuning jobs, evaluation pipelines, and speculative decoding draft model training workflows.

How It Works

• Three-tier capture architecture: Tier 1 captures at the SageMaker AI Runtime API boundary (full input/output payloads, compatible with Model Monitor); Tier 2 captures ALB access logs including request paths, client IPs, and latencies; Tier 3 captures at the model pod container level with the deepest visibility, configurable buffering, sampling, and payload size limits.
• Independent tier enablement: Each tier can be enabled or disabled independently via the dataCapture section in the InferenceEndpointConfig or JumpStartModel CRD, allowing any combination of tiers to run simultaneously.
• Asynchronous, non-blocking delivery: Captured data is written asynchronously to a customer-specified Amazon S3 bucket, ensuring inference latency is not impacted by the capture process.
• Configurable sampling: Customers can configure sampling rates to control the volume of captured traffic, reducing storage costs while maintaining statistical representativeness.
• Customer-managed encryption: AWS KMS encryption with customer-managed keys is supported at Tier 1 and Tier 3, ensuring data-at-rest security compliance.
• Deterministic S3 path routing: Each deployment generates a unique, deterministic S3 prefix derived from a hash of the cluster ARN, namespace, CRD type, and deployment name, so repeated deployments consistently route to the same S3 subfolder.
• HyperPod Inference Operator integration: Data capture is enabled declaratively through the Kubernetes CRD, fitting naturally into existing GitOps and infrastructure-as-code workflows on EKS-orchestrated HyperPod clusters.

Why It's Important

• Eliminates custom logging overhead: Teams previously had to build and maintain bespoke sidecar containers or external logging pipelines to capture inference traffic; this is now a first-class, managed capability within HyperPod.
• Enables production-grounded model improvement: Real production traffic can directly feed fine-tuning jobs, closing the loop between deployment and continuous model improvement without manual data collection steps.
• Improves speculative decoding performance: Draft models for speculative decoding trained on actual production traffic distributions outperform generic draft models, directly improving inference throughput and latency for generative AI workloads.
• Supports compliance and auditability: Persistent, encrypted audit trails of model inputs and outputs satisfy regulatory and governance requirements for AI systems in regulated industries.
• Accelerates evaluation pipeline construction: Captured production data provides a realistic, continuously refreshed ground-truth corpus for building and running model evaluation pipelines without synthetic data approximations.
• Integrates with existing MLOps tooling: Native compatibility with SageMaker Model Monitor means drift detection and data quality monitoring workflows can be activated immediately on captured data without additional transformation.

How It's Different

• Multi-tier capture vs. single-point logging: Unlike typical inference logging solutions that capture at only one layer, HyperPod data capture offers three distinct capture points in the same request pipeline, giving teams flexibility to trade off payload fidelity, metadata richness, and infrastructure dependency.
• Managed vs. DIY pipelines: Previously, HyperPod customers had no native data capture option and were required to instrument their own logging infrastructure; this feature removes that undifferentiated heavy lifting entirely.
• Tier 3 works without SageMaker endpoint registration: Unlike Tier 1, which requires endpoint registration for Model Monitor compatibility, Tier 3 operates independently at the pod level, making it usable in deployments that bypass the SageMaker endpoint abstraction.
• Deterministic S3 path management: The hash-based S3 prefix scheme ensures idempotent data routing across CRD resubmissions, a design advantage over ad-hoc logging solutions that may scatter data across inconsistent paths.
• Non-blocking asynchronous design: The asynchronous delivery model is purpose-built to avoid adding latency to the inference path, a critical distinction for latency-sensitive generative AI serving workloads.

When to Prefer It

• Generative AI production deployments requiring drift monitoring: Use data capture when you need to detect input distribution shifts or output quality degradation over time using SageMaker Model Monitor on live traffic.
• Speculative decoding optimization: Use Tier 3 capture to collect production traffic for training custom draft models that match your actual workload distribution, improving token acceptance rates beyond what generic draft models achieve.
• Compliance-sensitive AI applications: Enable Tier 1 with KMS encryption when you need tamper-evident, encrypted audit logs of all model inputs and outputs for regulatory or legal requirements.
• Active fine-tuning pipelines: Use data capture to continuously harvest real-world inputs for periodic fine-tuning jobs, particularly when production data distribution evolves over time and retraining on fresh data is critical.
• Debugging and troubleshooting production issues: Enable Tier 2 (ALB) alongside Tier 3 to correlate network-level metadata (latencies, client IPs) with model-level payloads when diagnosing production inference anomalies.
• Teams running EKS-orchestrated HyperPod clusters: This feature is specifically suited for organizations already using the HyperPod EKS orchestrator and the Inference Operator, where CRD-based configuration aligns with existing Kubernetes-native workflows.

Availability

• General Availability: The feature is generally available as of May 20, 2026, with no preview or beta designation indicated.
• Orchestrator requirement: Only available for SageMaker HyperPod clusters using the EKS orchestrator; Slurm-orchestrated HyperPod clusters are not supported.
• Regional availability: Supported in all AWS Regions where Amazon SageMaker HyperPod is generally available.
• S3 dependency: Requires an accessible Amazon S3 bucket for data delivery; if no s3Uri is specified, data defaults to the TLS certificate bucket under a /data-capture/ prefix.
• Pricing: No separate pricing is explicitly announced for the data capture feature itself; standard Amazon S3 storage and request costs apply for captured data, and AWS KMS charges apply when customer-managed keys are used.
• Encryption support: Customer-managed AWS KMS encryption is supported at Tier 1 and Tier 3; Tier 2 (ALB access logs) does not include KMS encryption support per current documentation.

Related Resources

• https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-data-capture.html