AI Patches Security Flaws Correctly Only 26% of the Time, 1Password Study Finds
An internal evaluation by the security company 1Password found that AI coding tools produce flawed or incomplete security fixes more than half the time, and sometimes make things worse.
Key points
- 1Password tested 6,080 AI-generated security patches and found that only 26% fixed the flaw without any unintended side effects.
- AI tools produced what researchers call FLAWED fixes (code that looks correct but contains hidden defects) 53.9% of the time when the vulnerability was complex.
- Nearly half of all patches, 49.3%, left at least one exploitable attack path open.
- Two AI models were tested: ChatGPT-5.5 and Claude Opus 4.8, each costing roughly $2 to $3 per patch cycle.
- Anthropic, the company behind Claude, told 1Password that human experts should remain the final reviewers at current AI capability levels.
Software companies are under constant pressure to fix security flaws quickly. AI coding assistants promise to speed that up. A new study from 1Password, the password-management company, suggests the speed comes with a serious catch.
What did 1Password actually find?
More than half of AI-generated patches contained hidden defects, and fewer than one in four fixed the problem cleanly. The team evaluated 6,080 patches generated by two leading AI models against six real, recently disclosed vulnerabilities, which are publicly catalogued security flaws assigned identifiers like CVE-2026-45185.
| CVE / Identifier | What it affects | Risk type |
|---|---|---|
| CVE-2026-31431 | Copy operation flaw | Data integrity |
| CVE-2026-34197 | Apache ActiveMQ | Remote code execution |
| CVE-2026-45185 | EXIM mail server | Remote code execution |
| CVE-2026-22738 | Spring AI framework | Remote code execution |
| GHSA-wpqr-6v78-jr5g | Gemini CLI tool | Remote code execution |
| CVE-2026-8512 | Undisclosed component | Security flaw |
Researchers labelled bad outputs "FLAWED", short for Fix-Like Artifacts With Embedded Defects. The name captures the core problem: the code compiles, passes basic tests, and looks fine, but it is not actually secure.
Why do the patches fail?
The AI is solving the immediate test, not the underlying problem. Researcher Keith Hoodlet gave a concrete example from the Spring AI vulnerability. Both ChatGPT-5.5 and Claude Opus 4.8 generated patches that filtered out the specific input characters used in the test exploit, a proof-of-concept attack, rather than removing the root cause. If an attacker used slightly different inputs, the original flaw would resurface.
More than a third of patches that initially looked successful were classified as "fragile" for exactly this reason.
Beyond fragility, 2.3% of patches fixed the original flaw but introduced a brand-new security weakness. Another 2.2% managed both: they failed to fix the original issue and added a new one.
Should security teams stop using AI for patches?
No, but they should not treat AI output as finished work. Anthropic, the company that makes Claude, told 1Password that human domain experts must remain the final reviewers given where AI capability sits today. That view is consistent with how most CTI analysts, those who track and analyse cyber threats professionally, already approach AI-assisted work: as a first draft, not a final product.
Cost is also a factor that cuts both ways. Each patch-and-check cycle cost roughly $2.11 with ChatGPT-5.5 and $2.81 with Claude Opus 4.8, reported first by CSO Online. Hoodlet's argument is that the real expense is the expert time required to verify whether any given AI patch is actually safe to ship.
For ordinary users, the practical concern is this: software you rely on every day may be patched with AI-generated fixes. The research suggests those fixes deserve the same careful human review that a handwritten patch would get. Demand for that review does not disappear just because a machine wrote the code.
