AWS Benchmarks AI Agents on Live Cloud Tasks, Not Static Fixtures
AWS's open-source aws-bench scores AI agents on real cloud work: fixing misconfigurations, provisioning infrastructure, and running live environments. Each scenario is deployed into a disposable, real AWS account via CDK stacks, then graded by an LLM judge or a check against live AWS state. There are no published scores or a leaderboard yet, just the harness. Engineering teams can extend scenarios to their own stack.