OpenAI builds AI hacker to harden models, refuses to release it
OpenAI has built a highly effective AI system designed to hack its own models, a move that exposes the growing security risks for European businesses deploying autonomous AI agents.
OpenAI has built an automated hacker designed to break its own artificial intelligence systems, but the company is keeping the tool locked away because it is too dangerous to release. The model, called GPT-Red, hunts for ways to hijack or sabotage AI systems, automating security work that humans have traditionally done by hand.
The tool represents a shift in how AI safety is managed, moving from manual testing to machine-speed automated attacks. OpenAI put GPT-Red in a self-play loop against defender models, pouring in what it calls unprecedented computing power for safety work. As defenders learned to block attacks, GPT-Red was forced to invent more sophisticated methods.
The system proved highly adept at finding prompt injection flaws, where hidden instructions trick a model into breaking its rules. It even discovered a new class of attack called "fake chain of thought," which plants false information in a model's private working memory. “It’s like if I told you that 1+1=3 and that you have verified this already,” said OpenAI researcher Chris Choquette-Choo. “The model’s like, ‘Oh, okay, of course,’ and it just spits out 3.”
For businesses integrating AI into physical or digital operations, the tests underscore concrete financial risks. GPT-Red successfully attacked an AI agent controlling a real office vending machine, cutting prices to the 50-cent minimum and cancelling customer orders. OpenAI says it has disclosed these flaws.
The testing results highlight the scale of the threat autonomous systems pose to each other. In a rerun of a 2025 test, GPT-Red cracked 84% of scenarios, compared to just 13% for human red-teamers. Against an older GPT-5 model, over 90% of GPT-Red's strongest attacks succeeded.
OpenAI used GPT-Red to train its newest model, GPT-5.6, reducing the attacker's success rate to under 23%. However, the decision to withhold GPT-Red establishes a critical precedent for the industry. The computing power required to build such a tool means AI safety is becoming an expensive, high-barrier arms race, raising costs for any company trying to compete.
The tool still has blind spots, struggling with prolonged back-and-forth attacks and instructions hidden inside images. Human testers also continue to catch flaws it misses. “I think human expertise will still be very important,” said Jessica Ji, an AI security analyst at Georgetown’s CSET. A full paper on the project is due later this week.
