AI Assistants Are Inventing Fake Web Addresses — and Criminals Are Buying Them Up

Key points

• Palo Alto Networks' Unit 42 published research on 30 June identifying 250,000 fake web addresses invented by AI tools across 685,339 queries to two AI models.
• The 913 brands studied also had 13,220 confirmed malicious URLs already circulating alongside the AI-generated ones.
• In one documented case, criminals registered a flagged fake address 23 days after researchers first spotted it, then used it to steal login credentials.
• The criminals used an AI coding assistant to build the entire fraud kit, including copying real shop pages and setting up a hidden channel to collect stolen data.
• Security researchers warn the threat could eventually reach a point where no human click is needed, as AI systems act on bad recommendations automatically.

When you ask an AI assistant for help finding a company's website or integrating a software tool, it sometimes invents a web address that sounds completely plausible but does not actually exist. Researchers call this "hallucination," the tendency of large language models (AI systems trained on enormous amounts of text) to confidently produce wrong answers.

Criminals have spotted the pattern.

A report from Palo Alto Networks' Unit 42 security research team, published 30 June, shows attackers are now systematically querying AI tools to find the fake addresses those tools generate most often, then registering those addresses as real websites. Phantom squatting is the name for that tactic, and we first covered it on 1 July when Unit 42 documented the pre-positioning mechanics.

How does phantom squatting actually work?

Criminals probe AI tools with questions about real brands, note which invented web addresses appear repeatedly, register those addresses for a few dollars each, then park malicious content there and wait for someone, or something, to follow the AI's recommendation.

"The attack chain is simple: probe models for invented domains that appear repeatedly, register the most useful names, place phishing or malicious content behind them, and wait for a person (or, increasingly, an autonomous agent) to follow the recommendation," Johan Edholm, a security engineer and co-founder at web-security firm Detectify, told Dark Reading. "It's cheap and scalable, which is what actually makes an attack dangerous."

The tactic resembles typosquatting, where criminals register addresses like "gooogle.com" to catch people who mis-type popular sites. Phantom squatting skips the typo entirely. It waits for an AI to invent a convincing fake, then directs users there directly, which puts it outside the watchlists security teams use to catch slight variations on known brand names.

What does a real phantom squatting attack look like?

Unit 42 tracked one case from start to finish. Researchers flagged a fake postal-service shopping address as high-risk. Twenty-three days later, criminals registered it. Behind it they had already built a complete phishing kit, a convincing copy of a real online shop designed to steal usernames and passwords. They used an AI coding assistant to scrape the genuine storefront, build the fake site's backend, and set up a Telegram channel (a messaging app) to quietly receive stolen credentials. The kit was called "Montana Empire."

Both the researchers and the criminals arrived at the same domain through the same mechanism: asking an AI what a postal service's shopping site would probably be called.

Should you worry about AI systems acting without human oversight?

The scale problem is real. AI assistants now sit inside company software and developer environments, so a bad recommendation no longer needs a human to click a link. An automated system could follow the AI's advice and send real data to a criminal's server without anyone noticing. Edholm describes this as the point of failure shifting from a person following bad advice to a system acting on it on their behalf.

What organisations should do now

Edholm's practical advice: verify any web address an AI recommends against official documentation before acting on it; restrict AI tools from connecting freely to addresses they haven't been explicitly approved to reach; tightly limit what data those systems can access; and treat a confident-sounding recommendation as a prompt to check, not a green light.

For staff who use AI assistants day to day, the same principle applies. If an AI gives you a web address you haven't seen before, look it up independently before you log in or share anything.

The uncomfortable truth here is that the AI didn't need to be compromised for any of this to work. It just needed to be wrong.