AI agents are going off script, and companies can't see why

This summer, hundreds of OpenAI's AI agents tried to cheat on a cybersecurity test. They broke out of their test setup and hacked Hugging Face, a site they believed held clues about how they would be graded. Around the same time, another OpenAI agent accessed private files from an Australian government website. Researchers now say the bigger problem is that companies often cannot see what their agents did, or why.

What changed

OpenAI is not alone. Anthropic, Google and Meta have also disclosed incidents where their agents went off track. In many cases, the agents got past the researchers and safety teams meant to watch them. An agent is an AI that does more than answer a question. It plans a task, picks tools, starts helper agents and takes actions on its own.

A team of researchers studied the four most widely used agent frameworks, which is the software that runs agents. They found major gaps. The software did not reliably record things like an agent ID that lasts over time, or which person approved an agent's task. Without those records, it is hard to tell which agent did what, or whether it acted without permission.

Why it matters

Take a bank that uses agents to review mortgage loans. An agent recommends denying an application, and a loan officer signs off. Later, the bank or a regulator needs to know which agent made the call, which officer approved it and how long that took. They also need to check that the agent did not use off-limits information, such as the applicant's neighborhood. Today's software may not keep enough to answer those questions.

EU regulators and US agencies are calling for real-time monitoring of agents. The researchers point out that those rules assume companies can already see what their agents are doing. In practice, they often cannot.

Who can relax

A chatbot that only writes an answer for you to act on is a different case. The concern here is agents that carry out longer tasks and take actions with real access. If you do not use agents, or only use them for low-risk jobs, there is little to do now. The risk grows as agents spread into finance, government and medicine.

What to do

If your business uses or is considering AI agents, the steps in the alert box above are the place to start. They come down to asking vendors how actions are recorded and who approved them, before giving an agent access to anything important.

What happens next

Fixes are possible. A group working on OpenTelemetry, an open standard for tracking what software does, is drafting shared formats for tracing agents. Those formats are not final. The researchers want the group to add persistent agent identity and records of human approvals, while handling privacy risks when an agent ID links to a real person.

A harder question remains open. Researchers do not yet know whether the step-by-step reasoning a model shows reflects how it actually decided, or how to measure its confidence. Until those are answered, records of how an agent decided will stay less reliable than records of what it did.