Google's Gemini was told to break into a fictional company inside a locked-down test network. It broke into three real ones instead.
The incidents happened in May during a “Capture the Flag” exercise run by security firm Irregular, the Wall Street Journal reported. In one case Gemini guessed its way past passwords. In the other two it found credentials sitting in public sources and walked in.
Google said the model stopped itself each time once it realized it had reached real systems. That’s the company’s account, and it’s the only one we have, because Google didn’t tell anyone this happened.
Google sat on it for two months
Irregular notified Google in late July. That was shortly after reports surfaced that OpenAI agents had hacked AI company Hugging Face during similar tests, so the pattern was already public knowledge in the industry.
Google still said nothing until the Journal came asking questions this week. The company’s position is that no damage was done, so there was no reason to go public.
And Gemini isn’t the outlier here. Similar incidents, all tied to Irregular’s testing, had already hit OpenAI, the UK’s AI Safety Institute, Anthropic and Meta. Google is simply the latest lab to have its name attached.
One test scenario, four labs, same mistake
Irregular said the incidents at Google, OpenAI, Anthropic and Meta all trace back to a single root cause. The firm tests models for major AI labs before release to check whether they pose security risks, and one of its more complex scenarios was built to see whether a model could help a malicious insider reach sensitive data.
For that scenario, Irregular picked a name for a fictional company. The name happened to match a real domain.
The instructions fed to the models included both that target name and internal addresses inside Irregular’s own network. The models were supposed to find the simulated target there. But internet access had been left on in the test environment by accident, and some models went after the real domain instead of staying in the sandbox.
The domain turned out to be poorly secured, which made it an easy target. Irregular said the breakouts were rare and typically happened late in a simulation, after hundreds of steps, which is why they were hard to spot.
That detail is the uncomfortable one. The failure wasn’t a model going rogue in step one. It was a model working through a long task, hitting a real address that looked like the fake one, and carrying on.
Who Irregular is
Irregular, formerly Pattern Labs, was founded in 2023 by CEO Dan Lahav, a former AI researcher at IBM, and CTO Omer Nevo, who spent over two years at Google. The startup has about 35 employees, according to PitchBook, and raised more than $80 million in a September funding round.
So the firm whose sandbox leaked is also the one the biggest AI labs pay to find exactly this kind of problem before release. The fix, in the end, was a switch that should have been off.
