AI Patches Security Flaws Correctly Only 26% of the Time, 1Password Study Finds
An internal evaluation by the security company 1Password found that AI coding tools produce flawed or incomplete security fixes more than half the time, and sometimes make things worse.
Key points
- 1Password tested 6,080 AI-generated security patches and found only 26% fixed the flaw without unintended side effects.
- AI tools produced what researchers call FLAWED fixes (code that looks correct but contains hidden defects) 53.9% of the time when the vulnerability was complex.
- Nearly half of all patches, 49.3%, left at least one exploitable attack path open.
- Two AI models were tested: ChatGPT-5.5 and Claude Opus 4.8, each costing roughly $2 to $3 per patch cycle.
- Anthropic, the company behind Claude, recommended keeping human domain experts as final reviewers at current AI capability levels.
Software companies are under constant pressure to fix security flaws quickly. AI coding assistants promise to speed that up. A new study from 1Password, the password-management company, suggests the speed comes with a serious catch.
What did 1Password actually find?
Fewer than one in four AI-generated patches fixed the problem cleanly, and more than half contained hidden defects. The team evaluated 6,080 patches generated by two leading AI models against six real, recently disclosed vulnerabilities, which are publicly catalogued security flaws assigned identifiers like CVE numbers (we explained how that system works in our 23 July story).
| CVE / Identifier | What it affects | Risk type |
|---|---|---|
| CVE-2026-31431 | Copy operation flaw | Data integrity |
| CVE-2026-34197 | Apache ActiveMQ | Remote code execution |
| CVE-2026-45185 | EXIM mail server | Remote code execution |
| CVE-2026-22738 | Spring AI framework | Remote code execution |
| GHSA-wpqr-6v78-jr5g | Gemini CLI tool | Remote code execution |
| CVE-2026-8512 | Undisclosed component | Security flaw |
Researchers labelled bad outputs "FLAWED", short for Fix-Like Artifacts With Embedded Defects. The code compiles and looks fine. It isn't actually secure.
Why do the patches fail?
The AI solves the immediate test, not the underlying problem. Researcher Keith Hoodlet gave a concrete example from the Spring AI vulnerability: both ChatGPT-5.5 and Claude Opus 4.8 generated patches that filtered out the specific input characters used in the proof-of-concept exploit rather than removing the root cause. Use slightly different inputs, and the original flaw resurfaces.
More than a third of patches that initially looked successful were classified as "fragile" for exactly this reason. Beyond fragility, 2.3% of patches fixed the original flaw but introduced a new security weakness. Another 2.2% managed both: they failed to fix the original issue and added a fresh one.
A separate finding worth tracking: 20.1% of patches resolved the vulnerability but altered application behaviour in the process, meaning they weren't clean fixes either.
Should security teams stop using AI for patches?
No, but treating AI output as finished work is a mistake. Anthropic told 1Password, as reported by CSO Online, that patch generation has outpaced patch verification and that domain experts must remain final reviewers at current model capabilities. That's consistent with how most CTI analysts (professionals who track and assess cyber threats) already approach AI-assisted work: first draft, not finished product.
Cost also cuts both ways. Each patch-and-check cycle ran roughly $2.11 with ChatGPT-5.5 and $2.81 with Claude Opus 4.8. Hoodlet's argument is that the real expense is the expert time required to verify whether any given patch is safe to ship.
For ordinary users, the practical concern is direct: software you rely on may be patched with AI-generated fixes that passed automated tests but left attack paths open. That research result doesn't change just because the code was written in seconds instead of hours.
