Amazon SageMaker AI Announces New observability capability For Inference Endpoints
Get instant LLM inference visibility — TTFT, GPU health, KV cache, and autoscaling — in one dashboard with zero instrumentation.
View original announcement →Visual Summary
graph TD
A{{SageMaker AI Inference Observability}}:::announced
B(SageMaker AI Endpoints):::compute
C(Amazon CloudWatch):::storage
D([OTel Collector]):::feature
E([Insights Dashboard]):::feature
F(Amazon Managed Grafana):::compute
G([GPU/Node Metrics]):::feature
H([Token Performance Metrics]):::feature
I((Platform Teams)):::external
I ==>|"monitors"| A
A -->|"scrapes"| B
B -->|"exposes"| G
B -->|"exposes"| H
D -->|"collects"| G
D -->|"collects"| H
D -->|"publishes via OTLP"| C
C -->|"renders"| E
C -.->|"PromQL endpoint"| F
I -->|"views"| E
I -.->|"queries"| F
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon SageMaker AI has launched a detailed observability capability for real-time inference endpoints, providing comprehensive, out-of-the-box visibility into generative AI inference workloads. The feature surfaces token-level performance metrics (TTFT, inter-token latency, tokens per second), GPU health, KV cache utilization, inference component placement, and autoscaling behavior in a unified view. A pre-built SageMaker AI Insights dashboard in Amazon CloudWatch and native PromQL support for Grafana eliminate the need for manual metric correlation or custom instrumentation.
How It Works
- OpenTelemetry-native collection: An embedded OTel Collector scrapes Prometheus endpoints from DCGM (GPU metrics via NVIDIA DCGM exporter), node exporters (CPU, memory, disk), and inference framework containers (vLLM, SGLang), then publishes metrics to Amazon CloudWatch via OTLP.
- Rich dimensional labeling: Every metric is tagged with labels including
aws.sagemaker.endpoint.name,aws.sagemaker.inference_component.name, instance ID, availability zone, and instance type, enabling precise filtering and aggregation without additional tagging effort. - Per-GPU and per-inference-component attribution: DCGM metrics include per-inference-component attribution on multi-tenant instances, so operators can identify exactly which model is consuming GPU resources.
- Inference framework metrics (no instrumentation required): Native vLLM and SGLang metrics — TTFT, inter-token latency (ITL), KV cache utilization, queue depth, batch size, tokens per second, and concurrent requests — are collected and published automatically.
- Pre-built CloudWatch Insights dashboard: A single-pane dashboard aggregates token latency, GPU utilization, inference component copy counts, scaling events, and cold start breakdowns, with no manual dashboard construction needed.
- PromQL query support: Metrics are queryable using PromQL syntax in CloudWatch, CloudWatch Query Studio, or Amazon Managed Grafana via a regional PromQL endpoint, with a pre-configured Grafana dashboard template available for import.
- Configurable scrape frequency: The
MetricPublishFrequencyInSecondsparameter accepts values of 10, 30, 60, 120, 180, 240, or 300 seconds (default: 60s); control-plane metrics (lifecycle, autoscaling, ICE diagnostics) are event-driven and unaffected by this setting.
Why It's Important
- Eliminates manual metric correlation: Previously, diagnosing a latency spike required manually searching CloudWatch for per-endpoint metrics and correlating them with GPU saturation or KV cache exhaustion — a process that could take hours. This capability surfaces all relevant signals in one place, reducing mean time to resolution to minutes.
- Closes the GenAI observability gap: Standard CloudWatch endpoint metrics do not expose token-level signals (TTFT, ITL, KV cache) critical for LLM inference quality. This feature fills that gap natively, without requiring customers to build custom metric pipelines.
- Enables proactive autoscaling tuning: By exposing queue depth, cold start breakdowns, and scaling event timelines alongside token latency, teams can identify and correct under-provisioning or misconfigured scaling policies before they impact end users.
- Supports multi-tenant GPU accountability: Per-inference-component GPU attribution on shared instances allows platform teams to enforce SLAs and chargeback costs per model, which is essential for organizations running multiple models on the same hardware.
- Reduces operational toil for platform teams: Zero-instrumentation, automatic metric publication means ML engineers and DevOps teams do not need to maintain sidecar exporters, custom Lambda functions, or bespoke dashboards to achieve production-grade observability.
How It's Different
- Zero instrumentation vs. custom pipelines: Unlike self-managed observability stacks (e.g., manually deploying DCGM exporters, Prometheus, and Grafana on EC2 or EKS), SageMaker detailed observability requires no customer-side instrumentation — the OTel Collector is embedded and managed by the service.
- Token-level metrics vs. request-level only: Standard SageMaker endpoint CloudWatch metrics report invocation counts and latency at the HTTP request level. This capability adds LLM-specific signals (TTFT, ITL, KV cache, TPS) that are invisible in the standard metrics namespace.
- Unified control-plane + data-plane view: Autoscaling events, inference component lifecycle, and ICE diagnostics are co-located with GPU and token metrics in the same dashboard, whereas previously these required separate CloudWatch queries across different namespaces.
- Native PromQL without a Prometheus backend: Metrics are stored natively in CloudWatch as OTel data and are queryable via PromQL syntax — customers get Prometheus-style query ergonomics without operating a Prometheus server or remote-write pipeline.
- Per-component GPU attribution on multi-tenant instances: Standard GPU monitoring tools report utilization at the instance level. This feature attributes GPU consumption to individual inference components, enabling model-level accountability on shared hardware.
When to Prefer It
- Running LLMs or generative AI models in production: Any team serving transformer-based models where TTFT and inter-token latency are user-facing SLOs should enable this to get immediate visibility into token-level degradation.
- Operating multi-model or multi-tenant inference endpoints: When multiple inference components share GPU instances, per-component attribution is essential for diagnosing which model is causing resource contention.
- Troubleshooting autoscaling lag or cold start latency: Teams experiencing slow scale-out or unexpectedly high cold start times can use the scaling event timeline and cold start breakdown panels to pinpoint configuration issues.
- Standardized on Grafana for observability: Organizations with existing Grafana infrastructure can connect directly via the PromQL endpoint and import the pre-configured template, avoiding dashboard duplication or context switching.
- Verifying availability zone compliance: Teams with AZ-pinning requirements or AZ-level SLAs can use the AZ-labeled metrics to confirm inference component placement and detect AZ imbalances.
- Optimizing KV cache and batch size configuration: Teams tuning vLLM or SGLang serving parameters (e.g.,
--max-model-len,--gpu-memory-utilization) can use KV cache utilization and batch size metrics to make data-driven configuration decisions.
Availability
- GA status: Generally available as of June 18, 2026; no preview or beta qualifier indicated in the announcement.
- Supported regions (17 total): US East (N. Virginia), US East (Ohio), US West (Oregon), US West (N. California), Canada (Central), South America (São Paulo), Europe (Ireland), Europe (Frankfurt), Europe (London), Europe (Stockholm), Europe (Zurich), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Seoul), and Asia Pacific (Jakarta).
- Pricing: Not explicitly stated in the announcement; customers should consult the SageMaker AI pricing page, as CloudWatch metric ingestion and storage costs may apply for OTel metrics published via OTLP.
- Supported inference frameworks: vLLM and SGLang are explicitly supported for inference framework metrics; other frameworks may not expose token-level metrics automatically.
- Scrape frequency constraint: The
MetricPublishFrequencyInSecondsparameter is limited to discrete values (10, 30, 60, 120, 180, 240, 300); arbitrary intervals are not supported. - PromQL is query-language only: No Prometheus server or Prometheus-compatible backend is involved; PromQL is supported solely as a query syntax against CloudWatch-stored OTel data.
Related Resources
- https://docs.aws.amazon.com/sagemaker/latest/dg/monitoring-cloudwatch-detailed-observability.html
- https://aws.amazon.com/sagemaker/ai/?trk=df5b8c0e-2f56-49fa-a37d-691eba3e8954&sc_channel=ps&trk=c5e59169-d233-4eef-b6e2-e3e751bfb3f2&ef_id=:G:s&s_kwcid=AL!4422!3!!e!!o!!amazon%20sagemaker%20ai!487441853!1151190521794397&msclkid=9745f631dad319a20f8e9c02eb1bda85
