AWS announces aws-bench, an open-source benchmark for AI agents on AWS
Now you can objectively score any AI agent on real AWS tasks — provisioning, troubleshooting, and diagnosis — using live cloud environments.
View original announcement →Visual Summary
graph TD
A{{aws-bench}}:::announced
B((AI Agents)):::external
C([CLI Tooling]):::feature
D(AWS CDK Stacks):::compute
E([Sandboxed Container]):::feature
F([LLM Judge Scoring]):::feature
G(Disposable AWS Accounts):::compute
H(aws-bench-datasets):::storage
I(Harbor Framework):::compute
B ==>|"evaluated by"| A
A -->|"manages lifecycle"| C
C -->|"provisions"| D
D -->|"deploys to"| G
A -->|"loads tasks"| H
B -->|"executes in"| E
E -->|"scoped access"| G
A -->|"verifies results"| F
A -.->|"extends"| I
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
AWS has released aws-bench, an open-source benchmark in research preview that objectively measures how accurately and efficiently AI agents complete real-world AWS tasks. The benchmark provides a curated suite of test cases derived from actual AWS usage patterns, covering investigation, troubleshooting, and infrastructure creation scenarios. Each test case pairs a natural-language query with a defined cloud resource state and a ground-truth answer, enabling consistent, reproducible scoring of any agent or foundation model.
How It Works
- Environment Provisioning: The CLI tool (
aws-bench env init+env setup) creates disposable, isolated AWS accounts and deploys scenario-specific infrastructure defined as CDK stacks, ensuring each test run starts from a clean, known state. - Agent Execution: The agent under test runs inside a sandboxed container with scoped AWS credentials, interacting with real live AWS infrastructure rather than static mocks or fixtures.
- Scoring: Results are verified using either an LLM judge for read-only diagnosis tasks or a programmatic check against live AWS state for tasks that create or modify resources, providing faithful and automated evaluation.
- Task Dataset: Test cases are maintained in a companion repository (
aws-bench-datasets) and cover three primary task categories: diagnosing misconfigurations, provisioning infrastructure, and operating live cloud environments. - Built on Harbor: aws-bench extends Harbor, an open-source agent/LLM evaluation framework, adding AWS-specific environment provisioning, scenario definitions, and verifiers.
- CLI Tooling: A unified CLI handles the full lifecycle — instantiating test environments, executing evaluation runs, scoring results, and resetting resource state between runs.
Why It's Important
- Objective Measurement Gap Filled: Until now, there was no standardized, public benchmark for measuring AI agent performance on AWS-specific tasks, making it difficult to compare models or track progress systematically.
- Grounded in Real Usage: Test cases are derived from analysis of actual AWS usage patterns, meaning benchmark scores correlate with real-world practitioner value rather than synthetic or academic scenarios.
- Accelerates Model Improvement: Foundation model providers and agent framework developers gain a reproducible signal they can use to fine-tune models, improve tool-use, and iterate on agent harnesses specifically for cloud operations work.
- Reproducibility and Trust: By running against real, disposable AWS environments with programmatic or LLM-judged verification, aws-bench produces results that are verifiable and not gameable through static fixture memorization.
- Community Benchmark Standard: As an open-source release on GitHub, it invites community contributions and has the potential to become an industry-standard leaderboard for cloud-native AI agents.
How It's Different
- Real Infrastructure, Not Mocks: Unlike most agent benchmarks that evaluate against static fixtures or simulated environments, aws-bench provisions and tears down actual AWS infrastructure for every test run.
- AWS-Specific Task Coverage: General-purpose agent benchmarks (e.g., SWE-bench, GAIA) do not cover cloud operations tasks; aws-bench is purpose-built for the AWS domain including IAM, networking, compute, and storage scenarios.
- Dual Scoring Modes: The combination of LLM-judge scoring for diagnostic tasks and programmatic live-state verification for mutating tasks provides more appropriate and accurate evaluation than a single scoring method.
- Derived from Real Usage: Task scenarios are sourced from analysis of real AWS customer usage patterns rather than hand-crafted hypotheticals, increasing ecological validity.
- Full Lifecycle CLI: The tooling manages environment setup, execution, scoring, and teardown in one integrated workflow, reducing friction compared to assembling evaluation pipelines from scratch.
- Built on an Extensible Framework: By extending Harbor, aws-bench inherits a general agent evaluation foundation while adding domain-specific layers, making it easier to contribute new scenarios or integrate new agent harnesses.
When to Prefer It
- Foundation Model Providers: Use aws-bench when evaluating or fine-tuning a model's ability to reason about AWS services, APIs, and cloud resource states as part of a model release process.
- Agent Framework Developers: Use it when benchmarking different agent architectures (e.g., ReAct, tool-use loops, multi-agent systems) against a consistent set of AWS operational tasks.
- Platform Engineering Teams: Use it to evaluate which model + agent combination is most reliable before deploying an AI assistant or automation agent into a production AWS environment.
- AI Researchers Studying Cloud Agents: Use it as a reproducible research artifact to publish results on agent performance for cloud operations, enabling apples-to-apples comparisons across papers.
- Regression Testing During Development: Use it as part of a CI/CD pipeline to detect performance regressions in an agent's AWS task-completion capability as the underlying model or tooling changes.
- Competitive Model Selection: Use it when selecting between competing foundation models for an AWS-focused agentic application and needing objective, task-specific performance data.
Availability
- Status: Research preview as of July 24, 2026; not yet generally available (GA).
- Access: Available now on GitHub at https://github.com/aws-bench/aws-bench with setup instructions in the README.
- Pricing: Open-source and free to use; however, running evaluations incurs standard AWS charges for the real infrastructure provisioned during test execution.
- Regions: Not explicitly specified; region availability is likely constrained by the AWS services used in individual CDK-defined scenarios and the user's configured AWS account access.
- Task Datasets: Maintained in a separate companion repository (
aws-bench-datasets); the breadth and coverage of scenarios may be limited during the research preview phase. - Limitations: Early-stage release with only 13 commits at launch; community adoption, dataset size, and scenario diversity are expected to grow over time.
