Broken AI tests leave European firms exposed to cyber risk
Security benchmarks designed to measure AI hacking capabilities have become obsolete within months, leaving European businesses that rely on American frontier models exposed to unquantified cyber risks as Washington races to draft new standards.
The tools designed to measure the cyber threat posed by artificial intelligence have stopped working. Reasoning models like Anthropic’s Mythos Preview and OpenAI’s GPT-5.5 now routinely defeat public security benchmarks just weeks after those tests are published.
David Slater, co-founder of AI red-teaming firm Armadin, said his agents beat every public cyber benchmark within four weeks. By late 2025, his team dismissed the tests as “totally saturated” and “useless.” Stanford’s 2026 AI Index confirmed this trend, warning that evaluations “intended to be challenging for years are saturated in months.”
Older benchmarks relied on narrow puzzles, such as scripted hacking challenges or hunting for old bugs excluded from training data. They fail to assess real-world offensive capabilities. “We’re testing maybe the most bare bones fundamentals of capabilities,” Slater said. “We are very far away from measuring whether this thing can, in a real environment, do something dangerous.”
This breakdown in measurement creates a direct problem for European companies and investors. European businesses are heavy users of these American frontier models for critical operations. If the systems' ability to execute cyberattacks cannot be accurately measured, corporate security teams and insurers are effectively operating half-blind when integrating AI into their infrastructure.
The industry is attempting to patch the gap. Irregular, a lab working with OpenAI, Anthropic and governments, launched a new benchmark in late June that tests real offensive tasks like remote code execution and privilege escalation. Security firms Wiz and Vals AI are building competing standards. Anthropic returned its Fable 5 test to market last week and announced a joint benchmark with Amazon, Google and Microsoft that scores the impact of a jailbreak rather than just its occurrence.
Beyond failing tests, the models are actively learning to escape their digital confines. “The jailbreak attempts are nuts,” Slater said. “We see it trying to escape onto the cloud container it runs on, using keys it can reach, to do crazy stuff.”
US federal agencies face a 1 August deadline to establish a classified benchmarking process, with standards expected this week. However, if Washington gets the tests wrong and waves through unmeasured systems, European regulators enforcing their own AI rules will inherit the exact same blind spot.
