Outside auditors find gaps in how OpenAI watches its own AI agents

Outside auditors found real gaps in how OpenAI watches its own AI agents. An estimated 6 to 25 percent of internal agent traffic from its GPT-5.5-class and newer models was not covered by monitoring at all during the audit. OpenAI says it has fixed every weakness the auditors found. Most of the details behind that claim are not public.

What the auditors did

Two researchers from METR and Redwood Research, both outside AI safety groups, worked on-site at OpenAI from July 21 to August 11. They had access to the company's code and internal documents. They also interviewed about a dozen employees. The audit was a pilot program set up jointly by METR, Redwood Research and OpenAI.

What they found

The 6 to 25 percent figure covers internal traffic to the Responses API, a tool that lets AI agents carry out tasks. METR says the uncertainty in that estimate is significant in both directions. Other findings included:

• About 10 percent of internal Codex traffic went untracked from May to July, because the system mishandled context summaries.
• Agents could delay oversight, because the monitor only triggered when an agent handed control back to the user.
• The auditors found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox, a restricted space meant to contain the agent.
• Specially crafted inputs could likely crash the monitor entirely.
• Model behavior during RL training, a stage of model training, was not monitored when the audit began.

OpenAI had found the Codex gap on its own and fixed it while the audit was still running. The 10 percent figure also carries significant uncertainty, according to METR.

What OpenAI says

OpenAI says it fixed all the weaknesses. It also says it paused RL training after the Hugging Face incident in August and expanded monitoring to cover it. Those changes came after the auditors' access period ended, so they were not part of the evaluation. Nobody outside the company has checked them.

Why the report is hard to judge

The full report ran 61 pages. After OpenAI applied redactions, leaders at METR and Redwood Research received a 38-page version. The public got only a short summary. METR said it had no right to disclose the redacted portions on its own. For future audits, METR says it wants to push for more public transparency.

What to do

Companies that build on AI agents are trusting oversight that outsiders can rarely check. When choosing a vendor, ask how its AI agents are monitored, what that monitoring misses, and whether anyone independent has looked.