What's New

Amazon Bedrock AgentCore has launched three new optimization capabilities in preview: recommendations, batch evaluations, and A/B tests. Together, these features close the "observe, evaluate, improve" loop for AI agents running in production, giving developers a systematic, data-driven path from identifying performance issues to deploying validated fixes. The capabilities are designed to help agents continuously improve over time as models evolve and user behavior shifts, without requiring ad-hoc developer intuition.

How It Works

• The optimization pipeline operates in three stages.
• First, the recommendations engine analyzes production traces and evaluation outputs already captured by AgentCore to automatically generate optimized system prompts and tool descriptions tailored to the specific workload — no manual prompt engineering required.
• Second, batch evaluations validate those recommendations offline by running them against pre-defined test case sets, providing a controlled quality gate before any change touches live traffic.
• Third, A/B tests expose the candidate changes to either a pre-defined test set or a slice of live production traffic, reporting statistical significance metrics so developers can make evidence-based promotion decisions.
• Critically, every recommendation requires explicit developer approval before it is deployed, preserving human oversight throughout the pipeline.

Why It's Important

• Agent quality degradation is a subtle but serious production risk: as foundation models are updated and user query distributions shift, an agent that performed well at launch can silently regress without any obvious failure signal.
• This toolchain gives teams a structured, repeatable mechanism to detect, validate, and ship improvements rather than relying on periodic manual audits or reactive firefighting.
• The statistical significance reporting on A/B tests is particularly valuable because it removes guesswork from promotion decisions, reducing the risk of shipping a change that improves aggregate metrics but harms specific user segments.
• For organizations operating agents at scale, this directly lowers the engineering overhead of maintaining agent quality over the full production lifecycle.

How It's Different

• Previously, AgentCore offered observability (production traces) and evaluations, but the path from an evaluation finding to an actual improvement was entirely manual — developers had to interpret results, hypothesize prompt changes, implement them, and re-run evaluations themselves.
• This new layer automates the hypothesis-generation step (recommendations) and adds two distinct validation stages (batch and live A/B) that were not previously available as first-party, integrated capabilities within AgentCore.
• Compared to general-purpose prompt optimization tools or external A/B testing frameworks, this solution is natively integrated with AgentCore's trace and evaluation data, meaning the recommendations are grounded in the agent's actual production behavior rather than synthetic benchmarks.

When to Prefer It

• These capabilities are best suited for teams running AI agents in production where agent quality is a measurable, ongoing concern rather than a one-time deployment task.
• They are especially valuable when agents handle diverse, evolving user workloads where prompt and tool-description tuning has a meaningful impact on output quality.
• Teams that lack dedicated prompt engineering resources will benefit most from the automated recommendations feature.
• The A/B testing capability is the right choice when the risk of a regression in production is high enough to warrant statistical validation before full rollout, as opposed to simply promoting changes based on offline batch evaluation scores alone.
• If an agent is still in early development or pre-production, the full optimization loop may be premature; the batch evaluation capability alone is useful at that stage.

Availability

• The optimization capabilities — recommendations, batch evaluations, and A/B tests — are currently in preview and are not yet generally available.
• They are accessible in all AWS Regions where AgentCore Evaluations is already available; the announcement does not enumerate specific regions, so users should consult the AgentCore documentation or the AWS Regional Services table for the current list.
• As a preview feature, it may be subject to service limits, API changes, and is typically not covered by production SLAs, so teams should plan accordingly before relying on it for critical workloads.