AI assistants fail at many routine cloud computing jobs, a new test finds

AI assistants still stumble on everyday cloud chores. A new open-source test called CloudOpsBench found that the best model solved 60 percent of its tasks. The runner-up solved 53 percent.

What the test covers

The cloud means rented computer systems that run a company's apps and data. DevOps is the work of keeping them running. CloudOpsBench is a benchmark, meaning a standard test for comparing tools. Version 1.0 has 45 tasks on AWS, built by experienced DevOps engineers. They include fixing services, removing resources, restricting access rights, rolling out updates and decommissioning projects.

Claude Fable 5.1 solved 60 percent. GPT-6 Astra managed 53 percent. The team, led by Ang Chen at the University of Michigan, says the main problem is not missing knowledge but weak reasoning.

Where the AI went wrong

• About 80 percent of failures came from misjudging the size of a task, skipping relevant services, or stopping too early.
• Roughly 20 percent were wrong actions, and some had consequences that cannot be reversed.

Every task runs the same way in the real AWS cloud and in an emulator, a practice copy. The project is on GitHub under the Apache 2.0 license, and a detailed failure analysis is available. Extensions for GCP, Azure and OCI are planned.

What is not known yet

The material does not say how other AI models score or when the extra cloud versions will arrive. For now the test covers only AWS. Any business letting AI manage its cloud should keep a person checking the work.