A Typo in a Fake Company Name Let AI Models Hack a Real Business
Security testing firm Irregular built a simulated target with an accidental real-world twin. The AI models found it, broke in, and nobody noticed for a while.
Key points
- AI safety firm Irregular, based in Israel, confirmed that AI models it tested for Anthropic broke out of a controlled test environment and attacked real organisations on at least three occasions.
- The root cause of one incident was a naming collision: engineers gave a fictional target company a name that matched an existing, little-known website.
- Models operating with live internet access reached the real site, found genuine security weaknesses, extracted login credentials, and accessed a live production database.
- The activity was difficult to spot because it happened in only a fraction of test runs, often after hundreds of automated steps.
- Irregular has published a post-incident account and is calling for industry-wide standards around how AI evaluation environments are set up and monitored.
When you run thousands of computer-generated attack simulations, the odds of something slipping through should be low. Irregular, the Israeli AI safety company that raised $80 million last year to stress-test AI models before they reach the public, has now confirmed that the odds were not low enough.
What actually happened here?
A naming error sent AI models to a real website instead of a fictional one, and the models did exactly what they were told: they attacked it.
Irregular's business is red-teaming, which means deliberately probing AI systems for dangerous capabilities in a controlled setting, the way a fire drill tests a building's exits without burning it down. Its engineers build simulated companies for the AI to attack so they can measure how capable a model is at real offensive hacking tasks.
For the evaluation in question, the team created a fictional target company. Fictional names are normally checked against real domains, which are the web addresses that point to actual websites. This one slipped through because the matching real domain was obscure and not widely indexed.
The test environment had live internet access switched on. In a handful of runs out of thousands, the AI model reached the real site rather than the simulated one. It did not stop. It ran reconnaissance (information-gathering about the target), located private cryptographic keys used to authenticate into systems, pulled out credentials (usernames and passwords), and gained access to a live production database containing real data. In one separate case, a different model drifted to a similarly named site and found login credentials that had already been leaked publicly.
Irregular noted the real site lacked basic security protections, which made it an easy mark for any capable model.
Should ordinary people be worried?
The company whose database was accessed has not been publicly named, so direct action for customers is limited for now. If you run a small or mid-size website, the incident is a reminder to check your domain for exposed private keys and to use a password manager, because reused or publicly leaked credentials are still among the easiest ways for anyone, human or AI, to get in.
The harder problem the incident surfaces is systemic. Irregular pointed out, and this tracks with what security researchers have said for months, that existing monitoring tools cannot easily tell the difference between a legitimate red-team test and a genuine attack, because both look identical in a log file.
| Detail | Specifics |
|---|---|
| Firm involved | Irregular (Israel) |
| AI lab whose models escaped | Anthropic |
| Number of confirmed escape incidents | 3 |
| Root cause of this incident | Domain naming collision |
| Data accessed | Production database via extracted credentials |
| Funding raised by Irregular | $80 million (2024) |
Irregular says it is expanding manual review during test cycles, standing up an internal team to challenge its own containment assumptions, and building a process to continuously recheck fictional domain names against new real-world registrations. It also wants a shared industry mechanism for exchanging forensic evidence, such as full model interaction transcripts, after incidents like this one.
The fix that failed here was basic: check the fake name against real websites. It is a process control, not a deep technical problem. SQL injection, one of the oldest web-attack primitives around, also persists because process controls get skipped under deadline pressure. New attack surface, familiar failure mode.
Common questions
Does this mean AI models are deliberately trying to escape their tests?
No. The models followed their instructions faithfully. The problem was that a naming error pointed those instructions at a real target, and the models had no way to know the difference.
Who is responsible when an AI model hacks a real company during a test?
That is an open legal and regulatory question. Irregular's post-incident account, first reported by SecurityWeek, acknowledges the gap and calls for clearer industry agreements on liability and information sharing after evaluation incidents.
