Amazon SageMaker AI now supports OpenAI-compatible APIs for inference endpoints
Swap one URL and your OpenAI SDK calls run on your own GPU instances inside your VPC — no rewrites, no SigV4, full control.
View original announcement →Visual Summary
graph TD
A{{SageMaker OpenAI-Compatible APIs}}:::announced
B((Developer Apps)):::external
C([OpenAI SDK / LangChain]):::feature
D([Bearer Token Auth]):::feature
E(SageMaker Endpoints):::compute
F([Multi-Model Hosting]):::feature
G(GPU Instances / VPC):::compute
H([Streaming Support]):::feature
I((Strands Agents)):::external
B ==>|"changes URL only"| C
C ==>|"chat/completions"| A
I -->|"invokes"| A
A -->|"authenticates"| D
A -->|"routes requests"| E
E -->|"runs on"| G
E -.->|"serves"| F
A -->|"passes through"| H
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon SageMaker AI now exposes an OpenAI-compatible API path (/openai/v1/chat/completions) on real-time inference endpoints, enabling developers to invoke SageMaker-hosted models using the OpenAI SDK, LangChain, Strands Agents, and any other OpenAI-compatible client. The only change required is swapping the endpoint URL — no custom integration code, SigV4 wrappers, or SDK rewrites are needed. This brings the familiar OpenAI developer experience to SageMaker's enterprise-grade infrastructure, including VPC isolation, custom GPU selection, and fine-tuned model support.
How It Works
- SageMaker endpoints now expose an
/openai/v1/chat/completionspath that accepts standard Chat Completions requests and returns responses directly from the container, including streaming support. - Authentication uses short-lived bearer tokens generated from existing AWS credentials via the
generate_tokenfunction in the SageMaker Python SDK; tokens are valid for up to 12 hours and requiresagemaker:CallWithBearerTokenandsagemaker:InvokeEndpointIAM permissions. - SageMaker routes requests based on the endpoint name embedded in the URL, so any OpenAI-compatible client resolves the correct backend without additional configuration.
- The feature is automatically enabled on all existing and new SageMaker real-time endpoints and inference components — no opt-in or redeployment is required.
- Multi-model deployments using inference components are fully supported, allowing multiple models (e.g., Llama, Mistral, a classifier) to be hosted on a single endpoint and addressed through the same OpenAI SDK interface.
- Streaming responses are passed through as-is from the container, preserving existing streaming logic in client applications without modification.
Why It's Important
- Eliminates the primary friction point for teams migrating from OpenAI or other hosted LLM APIs to self-managed infrastructure — the API contract stays identical, reducing migration risk and developer effort.
- Enables agentic frameworks (LangChain, Strands Agents, Vercel AI SDK, LLM gateways) to treat SageMaker endpoints as drop-in replacements for OpenAI, unlocking enterprise data-residency and compliance requirements without framework rewrites.
- Organizations can now run fine-tuned or open-source models on dedicated GPU instances inside their own VPC while preserving the same application code that previously called OpenAI's hosted API.
- Bearer token authentication eliminates the need to manage long-lived API keys or implement complex SigV4 signing in third-party tools, reducing operational overhead and security surface area.
- Multi-model hosting through inference components with a single unified interface reduces application-layer routing complexity and consolidates infrastructure costs.
How It's Different
- Unlike calling OpenAI's hosted API, SageMaker keeps all inference traffic within the customer's own AWS account and VPC, satisfying data-sovereignty, HIPAA, and other compliance requirements that prohibit sending data to third-party endpoints.
- Unlike previous SageMaker invocation (which required SigV4 signing or the Boto3 SDK), the new path uses standard bearer token authentication that any HTTP client or OpenAI-compatible framework can use natively.
- Unlike Amazon Bedrock's OpenAI-compatible layer, SageMaker's implementation supports fully custom or fine-tuned models stored in S3, not just AWS-curated foundation models.
- Unlike managed inference services (OpenAI, Azure OpenAI), SageMaker gives customers direct control over instance type, auto-scaling policy, and model weights, enabling cost and latency optimization not possible with black-box APIs.
- Unlike wrapping SageMaker with a proxy or middleware layer, this is a native platform capability with no additional hop, reducing latency and eliminating a failure point.
When to Prefer It
- When your team has existing applications or agentic pipelines built against the OpenAI SDK and you need to move inference in-house for data privacy, cost control, or regulatory compliance.
- When you are deploying fine-tuned or proprietary open-source models (e.g., domain-adapted Llama or Mistral variants) that are not available through any managed API provider.
- When you need predictable, dedicated GPU capacity with custom auto-scaling policies rather than shared, rate-limited hosted inference.
- When integrating SageMaker into an LLM gateway or routing layer (e.g., Bifrost, LiteLLM) that speaks the OpenAI Chat Completions protocol and cannot easily accommodate custom signing logic.
- When running multi-model workloads where different models serve different tasks and you want a single, consistent API surface across all of them without per-model client configuration.
- When operating in regions or industries where data must not leave a specific AWS region or VPC boundary, making third-party hosted APIs non-viable.
Availability
- Status: Generally Available (GA) as of May 21, 2026.
- Supported Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Europe (London), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Mumbai), Asia Pacific (Jakarta), South America (São Paulo), and Canada (Central).
- Pricing: No additional charge for the OpenAI-compatible API path itself; standard SageMaker real-time inference instance and data transfer costs apply.
- Supported Endpoint Types: Real-time inference endpoints and inference components; batch transform and asynchronous inference endpoints are not mentioned as supported.
- API Scope: Currently exposes the
/openai/v1/chat/completionspath (Chat Completions); other OpenAI API endpoints (embeddings, completions, images, etc.) are not listed as supported at launch. - Container Requirement: The underlying container must itself return OpenAI-compatible responses; SageMaker passes responses through as-is, so containers serving non-compatible formats will not automatically conform.
- Token Expiry: Bearer tokens have a maximum validity of 12 hours and a minimum of 1 second, configurable via the
expiryparameter ingenerate_token.
