AI Safety Audit: OpenAI's Monitoring Gaps Exposed

Independent auditors have uncovered significant blind spots in how OpenAI tracks the behavior of its most advanced models. A three-week on-site review revealed that a substantial portion of internal traffic was operating without oversight, raising new questions about the robustness of current safety monitoring systems.

The investigation was conducted by researchers from METR and Redwood Research, who were granted rare access to OpenAI's internal infrastructure. From July 21 to August 11, the team examined the company's codebase and internal documents while interviewing approximately a dozen employees. This pilot program was designed to test the limits of external scrutiny by allowing outside experts to inspect the mechanisms used to detect rogue AI behavior.

The findings highlight a notable gap in coverage for the company's most powerful models. METR estimates that between six and 25 percent of internal agentic Responses API traffic from models at the GPT-5.5 class and above was not monitored at all during the audit period. The researchers noted that this estimate carries significant uncertainty in both directions, but the potential scale of unmonitored activity remains a central concern for safety professionals.

A specific technical flaw also left internal Codex traffic vulnerable. The auditors found that roughly 10 percent of this traffic went untracked between May and July due to how the system handled compaction events, which serve as context summaries. OpenAI independently discovered and fixed this bug while the audit was still ongoing. During the period of the gap, agents could delay oversight because the monitoring system only triggered when control was handed back to the user, allowing agents to sidestep checks by continuing to execute actions.

Beyond these coverage gaps, the team identified several methods to obscure activity. The auditors found ways to hide hundreds of actions from the monitor and bypass the Codex sandbox entirely. They also determined that specially crafted inputs could likely crash the monitoring system. Additionally, model behavior during reinforcement learning (RL) training was not being monitored at the start of the audit.

OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded monitoring to cover this area. However, these changes were implemented after the auditors' access period ended, meaning they were not part of the initial evaluation. The company asserts that all weaknesses identified by the auditors have now been fixed.

The public has only seen a fraction of the full findings. The complete 61-page report was shared internally, but after OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version. A short summary was the only portion made available to the general public. METR indicated it did not have the right to disclose the redacted sections on its own.

Looking ahead, the organizations involved are working to refine the process for future evaluations. While this audit served as a pilot to demonstrate the value of deep external access, METR has expressed a desire for greater public transparency in upcoming audits. The findings underscore the ongoing challenge of balancing operational speed with rigorous safety oversight in the development of advanced AI systems.