Amazon Bedrock AgentCore introduces new optimization capabilities to continuously improve agents in production
Turn production agent traces into validated fixes automatically—closing the gap between silent failures and confident, data-proven improvements.
View original announcement →Visual Summary
graph TD
A{{AgentCore Optimization}}:::announced
B([Failure/Intent/Trajectory Insights]):::feature
C([AI Recommendations]):::feature
D([Batch Evaluation]):::feature
E([A/B Testing]):::feature
F((Production Agents)):::external
G(AgentCore Gateway):::compute
H([Configuration Bundles]):::feature
I((Development Teams)):::external
F ==>|"production traces"| A
A -->|"analyzes sessions"| B
B -->|"informs"| C
C -->|"generates changes"| H
H -->|"validates against"| D
D -->|"promotes to"| E
E -->|"splits traffic via"| G
G -->|"routes to"| F
I -.->|"reviews insights"| B
I -.->|"approves rollout"| E
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
AWS has launched new optimization capabilities in Amazon Bedrock AgentCore that create a closed-loop system for continuously improving AI agents using production trace data. The feature set includes three types of behavioral insights (failure, intent, and trajectory), AI-generated recommendations for system prompts and tool descriptions, batch evaluation, and A/B testing with live traffic splitting. These capabilities work across any agent runtime—AgentCore, AWS Lambda, Amazon EKS, or non-AWS environments—making them broadly applicable regardless of deployment architecture.
How It Works
- Failure Insights analyze hundreds of production sessions to surface recurring failure patterns, including silent behavioral failures that produce no error signal, rank them by user impact, and explain root causes so teams can prioritize fixes.
- Intent Insights cluster user requests by underlying goal, revealing what users are actually trying to accomplish across sessions—information that is invisible in per-trace or dashboard-level monitoring.
- Trajectory Insights group the execution paths agents take through tasks, identifying common patterns and statistical outliers that indicate unexpected or suboptimal agent behavior.
- Recommendations consume traces and evaluation outputs to generate specific, rationale-backed changes to system prompts and tool descriptions, grounded in observed production behavior rather than generic best practices.
- Configuration Bundles provide versioned, immutable snapshots of agent configuration (system prompts, model IDs, tool descriptions) that decouple behavioral changes from code deployments, enabling updates without redeployment.
- Batch Evaluation tests recommended changes against a customer-defined test dataset using multiple evaluators, reporting aggregate scores to catch regressions before any change reaches live users.
- A/B Testing splits live production traffic between two agent variants via AgentCore Gateway, scores each session with online evaluators, and reports statistical significance to confirm that improvements hold under real-world conditions before fleet-wide rollout.
Why It's Important
- Silent failures are the hardest to detect: Traditional monitoring catches errors but misses behavioral degradation—agents that respond confidently but incorrectly. This capability specifically targets that gap, surfacing failures that would otherwise only appear as customer complaints weeks later.
- Closes the production feedback loop: Most agent development workflows lack a systematic path from "something is wrong in production" to "here is a validated fix." AgentCore optimization provides that end-to-end loop without requiring custom tooling.
- Reduces reliance on manual prompt engineering: AI-generated recommendations grounded in actual trace data replace the slow, intuition-driven cycle of manually rewriting prompts and testing by hand.
- Statistical confidence before full rollout: A/B testing with live traffic provides evidence that a change works in production—not just on curated test sets—reducing the risk of regressions when rolling out updates fleet-wide.
- Framework and runtime agnostic: Because the optimization loop works with any agent runtime, teams are not locked into a specific deployment model to benefit from continuous improvement capabilities.
How It's Different
- Pattern-level analysis vs. trace-level review: Conventional observability tools require engineers to review traces one at a time. AgentCore optimization analyzes hundreds of sessions simultaneously to surface statistically significant patterns that no individual trace review would reveal.
- Data-grounded recommendations vs. generic prompt advice: Unlike general prompt engineering guides or LLM-assisted rewrites without context, recommendations are derived directly from the agent's own failure patterns and evaluation outputs, making each suggestion specific and immediately actionable.
- Production A/B testing vs. offline evaluation only: Most agent evaluation frameworks operate on static test datasets. AgentCore's A/B testing splits real production traffic and measures outcomes under live conditions, providing a fundamentally stronger signal about whether a change actually improves user experience.
- Behavioral change without redeployment: Configuration bundles decouple system prompt and tool description changes from code, allowing teams to iterate on agent behavior without triggering a full deployment pipeline—a capability not typically available in standard agent frameworks.
- Integrated improvement loop vs. stitched-together tools: The full cycle—observe, recommend, evaluate offline, validate in production—is available within a single service, eliminating the need to integrate separate monitoring, experimentation, and evaluation tools.
When to Prefer It
- When agents are in production but quality is hard to measure: Use failure and intent insights to establish a baseline understanding of what users are trying to do and where the agent is falling short, especially when error rates look normal but user satisfaction is unclear.
- When prompt changes are frequent but validation is ad hoc: Teams that iterate on system prompts without a structured evaluation process should use batch evaluation and recommendations to ensure changes improve quality rather than introduce regressions.
- When scaling an agent to a large user base: Before committing a configuration change to all users, use A/B testing to confirm improvements hold at scale under real traffic conditions rather than relying solely on test dataset performance.
- When silent behavioral failures are suspected: If customer feedback suggests the agent is giving wrong or unhelpful answers that don't appear as errors in logs, trajectory and failure insights are the appropriate tool to identify and diagnose the root cause.
- When agents are deployed outside of AgentCore Runtime: Teams running agents on Lambda, EKS, or non-AWS infrastructure can still benefit from the full optimization loop, making this relevant for hybrid or multi-cloud agent deployments.
- When multiple teams share responsibility for agent quality: Configuration bundles and versioned snapshots provide a structured way to manage and audit behavioral changes across teams without coupling agent configuration to application code releases.
Availability
- Failure, Intent, and Trajectory Insights: Available in preview today in 13 AWS Regions.
- Batch Evaluations, Recommendations, and A/B Testing: Generally available (GA) today in 14 AWS Regions.
- Supported GA Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Seoul), and Canada (Central).
- AgentCore Optimization (preview regions): Based on the regions table, the 13-region preview subset excludes some of the 14 GA regions; South America (São Paulo) and AWS GovCloud (US-West) are not listed for AgentCore optimization.
- Runtime compatibility: Works with AgentCore Runtime, AWS Lambda, Amazon EKS, and non-AWS environments.
- Pricing: Not explicitly stated in the announcement; refer to the Amazon Bedrock AgentCore pricing page for current details.
- Prerequisites: Optimization capabilities build on AgentCore Evaluations and AgentCore Observability, which must be configured; AgentCore Gateway is required for A/B testing traffic splitting.
