AI Assistants Are Inventing Fake Web Addresses — and Criminals Are Buying Them Up

Key points

• Palo Alto Networks' Unit 42 published research on 30 June identifying 250,000 fake web addresses invented by AI tools across 685,339 queries to two AI models.
• The 913 brands studied also had 13,220 confirmed malicious URLs already circulating alongside the AI-generated ones.
• In one documented case, criminals registered a flagged fake address 23 days after researchers first spotted it, then used it to steal login credentials.
• The criminals used an AI coding assistant to build the entire fraud kit — including copying real shop pages and setting up a hidden channel to collect stolen data.
• Security researchers warn the threat could eventually reach a point where no human click is needed at all, as AI systems act on bad recommendations automatically.

When you ask an AI assistant for help — finding a company's website, looking up a banking portal, or integrating a software tool — it sometimes invents a web address that sounds completely plausible but does not actually exist. Researchers call this "hallucination," the tendency of large language models (AI systems trained on enormous amounts of text) to confidently produce wrong answers.

Criminals have spotted the pattern.

A report from Palo Alto Networks' Unit 42 security research team, published 30 June, shows attackers are now systematically querying AI tools to find the fake addresses those tools generate most often, then registering those addresses as real websites. The tactic has a name: phantom squatting.

How does phantom squatting actually work?

The attack follows a straightforward four-step loop. Criminals probe AI tools with questions about real brands. They note which invented web addresses appear repeatedly. They register those addresses — a process that costs just a few dollars per domain. Then they park malicious content there and wait for someone, or something, to follow the AI's recommendation.

"It's cheap, repeatable, and scalable, which is what actually makes an attack dangerous," Johan Edholm, a security engineer and co-founder at web-security firm Detectify, told Dark Reading, which first reported the research.

The tactic is related to typosquatting — where criminals register addresses like "gooogle.com" to catch people who mis-type popular sites. The difference is that phantom squatting does not wait for a human to make a typo. It waits for an AI to invent a convincing fake, then directs users there directly.

That makes it harder to catch. Security teams normally watch for slight variations on known brand names. A freshly registered address that an AI invented from scratch sits outside those watchlists entirely.

Unit 42 tracked one case from start to finish. Researchers flagged a fake postal-service shopping address as high-risk. Twenty-three days later, criminals registered it. Behind it they had already built a complete phishing kit — a convincing copy of a real online shop designed to steal customers' usernames and passwords. They used an AI coding assistant to scrape the genuine storefront, build the fake site's backend, and set up a Telegram channel (a messaging app) to quietly receive stolen login details. The kit was called "Montana Empire."

Both the researchers who spotted the fake address and the criminals who exploited it arrived at the same domain through the same mechanism: asking an AI what a postal service's shopping site would probably be called.

The deeper worry is scale. Because AI assistants now sit inside company software, customer-service tools, and developer environments, a bad recommendation no longer needs a human to click a link. An automated system could follow the AI's advice and send real data to a criminal's server without anyone noticing.

What organisations should do now

Edholm's practical advice breaks down to four steps: verify any web address an AI recommends against official documentation before acting on it; restrict AI tools from connecting freely to addresses they have not been explicitly approved to reach; limit what data those AI systems can access; and treat an AI's confident-sounding recommendation as a starting point for a check, not as a green light.

For staff who use AI assistants day to day, the same principle applies: if an AI gives you a web address you have not seen before, look it up independently before you log in or share anything.