Anthropic's Mythos Preview Goes Bug-Hunting: What XBOW's Red Team Found

Anthropic quietly handed XBOW early access to a model it's calling Mythos Preview. XBOW, which runs an autonomous pentesting platform, put it to work on the kind of tasks that actually matter in offensive security: spotting vulnerabilities in source, reasoning about binaries, and confirming whether a bug is real on a live target.

The short version? Source-code analysis was where Mythos shone.

When pointed at codebases, the model surfaced a high volume of credible vulnerability candidates — including the unglamorous, hard-to-spot logic flaws that static analyzers routinely miss. Authentication and session-handling bugs sit squarely in that category. A SAML response that's signed but whose assertions aren't properly validated, an OIDC implementation that accepts tokens with the wrong aud claim, refresh-token rotation that forgets to revoke the old token: these are exactly the patterns that need a reader who understands intent, not just syntax.

XBOW's evaluation covered three workstreams.

For exploit discovery, Mythos was strong at generating candidate findings but, like every model in this space, leaned toward over-reporting. False positives are still the tax you pay for recall. Triage remains a human (or agent) job.

Reverse engineering was a weaker showing. Reasoning over stripped binaries, decompiled output, and the messy middle of IDA or Ghidra sessions is a different cognitive load than reading well-structured Python or Go. The model could describe what it saw. It struggled to chain that into working exploit logic.

Live-site validation — actually proving a bug exists on a running target — sat somewhere in the middle. Mythos could drive HTTP requests, interpret responses, and adjust. It occasionally invented endpoints that didn't exist, which is the LLM equivalent of confidently knocking on the wrong door.

A few things worth flagging for identity teams specifically.

First, source-aware models are getting genuinely good at finding authn/authz separation bugs. That's the class of flaw where the code checks who you are but forgets to check what you're allowed to do. Second, if your codebase is in a public repo or has been scraped, assume both offensive and defensive tooling will eventually point a Mythos-class model at it. Third, MFA wouldn't have helped any of the bug classes Mythos was best at finding — these are server-side logic flaws, not credential theft.

XBOW's full methodology and per-task results are in their writeup. Anthropic hasn't published a model card for Mythos Preview yet; when they do, expect the usual caveats about offensive-use evaluations and the responsible scaling policy.

The broader signal: frontier labs are now testing offensive capability with offensive vendors before release. That's a healthier loop than discovering it the hard way after launch. It also means defenders should be running the same models against their own code. If you don't, someone else will.