What's New

Amazon SageMaker AI now automatically caches container images on new instances during scale-out events, eliminating the need to pull large container images (10 GB+) from Amazon ECR each time a new instance launches. This capability delivers up to 2x faster end-to-end scaling for generative AI inference workloads by pre-pulling the container image so instances can begin serving traffic sooner. No configuration changes are required — SageMaker automatically caches whichever image URI is specified in the endpoint or inference component configuration.

How It Works

• Pre-caching mechanism: When an endpoint scales out and a new EC2 instance is provisioned, SageMaker proactively pre-pulls the container image from ECR onto the new instance before it is needed, so the image is already available locally when the container starts.
• Automatic detection: The service reads the image URI from the existing endpoint configuration or inference component definition and uses that to determine which image to cache — no additional user input or API changes are required.
• Parallel operations: Container image download and model artifact download from S3 normally happen in parallel; by eliminating the container pull step entirely, the overall critical path is shortened significantly.
• Supported endpoint types: The feature works with single-model endpoints (where each new instance hosts its own model copy) and inference component-based endpoints (where new instances are added to accommodate additional inference components).
• Accelerator instance support: The caching capability is specifically designed for and validated on GPU/accelerator instance types commonly used for generative AI inference workloads.
• Part of a broader suite: Container image caching complements two previously released features — sub-minute CloudWatch concurrency metrics (up to 6x faster load detection) and instance-store container caching (faster scaling on already-running instances) — forming a comprehensive scaling optimization stack.

Why It's Important

• Eliminates a major cold-start bottleneck: For generative AI workloads, container images from frameworks like vLLM, SageMaker LMI, and NVIDIA Triton routinely exceed 10 GB, meaning a full ECR pull could add several minutes of latency to every new instance launch during a traffic spike.
• Directly improves user-facing latency during traffic bursts: Faster scale-out means endpoints reach full capacity sooner, reducing the window during which requests may queue or experience elevated latency when demand suddenly increases.
• Zero operational overhead: Because the feature is fully automatic and requires no code or configuration changes, existing customers benefit immediately without any migration effort or re-deployment.
• Closes the gap for new-instance scenarios: Previous instance-store caching only helped when scaling could reuse already-running instances; this feature extends the benefit to the harder case where entirely new instances must be provisioned.
• Supports cost-efficient autoscaling strategies: With faster scale-out, teams can more aggressively scale down during low-traffic periods and rely on rapid scale-up when needed, improving cost efficiency without sacrificing responsiveness.

How It's Different

• vs. no caching (baseline): Previously, every new instance had to pull the full container image from ECR at launch time, adding minutes of cold-start latency; container image caching removes this step entirely for scale-out events.
• vs. instance-store container caching: The earlier instance-store caching feature only accelerates scaling operations that reuse existing, already-running instances; container image caching specifically targets the scenario where brand-new instances must be launched, covering the gap the prior solution could not address.
• vs. custom AMI baking: Some teams bake container images into custom AMIs to avoid pull latency, but this requires significant operational overhead to maintain and update; SageMaker's automatic caching achieves a similar outcome with no user effort and always reflects the current image URI in the endpoint config.
• vs. ECR pull-through cache: ECR pull-through caches reduce network egress costs and improve pull speeds from a registry perspective but still require the instance to perform a full image pull; SageMaker's approach eliminates the pull step on the instance entirely during scale-out.
• Integrated with the full scaling optimization suite: Unlike point solutions, this feature is natively integrated with SageMaker's sub-minute metrics and instance-store caching, providing compounding improvements across detection, provisioning, and image availability stages of the scaling lifecycle.

When to Prefer It

• Large generative AI models with heavy container dependencies: Workloads using frameworks like vLLM, SageMaker LMI, or NVIDIA Triton that produce container images of 10 GB or more will see the greatest absolute time savings from image caching.
• Endpoints subject to sudden or unpredictable traffic spikes: Applications where demand can surge rapidly (e.g., consumer-facing chatbots, batch inference APIs with bursty job submissions) benefit most from minimizing the time between a scale-out trigger and new instances becoming ready.
• Cost-optimized deployments that scale to near-zero: Teams that aggressively scale down to minimal instance counts during off-peak hours and rely on fast scale-up to handle returning traffic will find this feature critical for maintaining acceptable response times.
• Inference component-based endpoints requiring new instance provisioning: When inference component scaling cannot be satisfied by placing components on existing instances and must launch new ones, this feature ensures the container is ready immediately upon instance availability.
• Production SLAs with strict latency budgets: Any deployment with contractual or business requirements around p99 latency during scaling events should enable this feature to reduce the duration of the degraded-performance window.
• Multi-model or multi-framework environments: Endpoints that serve multiple model variants using different large container images benefit from per-image caching tied to the specific image URI in each configuration.

Availability

• General Availability: This feature is generally available as of June 30, 2026, with no preview or opt-in period required.
• Regional availability: Available in all AWS commercial regions where Amazon SageMaker Inference is supported; not available in AWS GovCloud or other non-commercial partitions at launch.
• No configuration required: The feature is automatically enabled for all qualifying endpoints — customers do not need to update endpoint configurations, IAM policies, or SDK versions to benefit.
• Supported instance types: Limited to accelerator (GPU/Inferentia/Trainium) instance types; standard CPU-only instance types are not explicitly listed as supported targets for this feature.
• Supported endpoint patterns: Works with single-model endpoints and inference component-based endpoints; other endpoint types may not be covered.
• Pricing: No additional charge is documented for the caching capability itself; standard SageMaker Inference instance and ECR data transfer pricing applies.

Related Resources

• https://aws.amazon.com/blogs/machine-learning/introducing-container-caching-in-amazon-sagemaker-ai-for-faster-model-scaling/