AI is getting better at breaking software than fixing it

Key points

• Veracode found that 44% of code written by AI contains at least one serious, well-known security flaw, based on tests of more than 100 model versions over the past year.
• No AI model tested by Veracode achieved a security pass rate above 68%, meaning every model produced unsafe code at least once in every three attempts.
• A separate study by 1Password's Off-By-1 Labs found that AI-generated fixes for known software vulnerabilities worked correctly only 26% of the time across more than 6,000 test runs.
• Researchers at Xint found 434 confirmed security flaws across 28 apps built or rebuilt by AI, including secrets accidentally baked into the code and runaway resource use.
• Better training data, not just more training, is the most plausible path to improvement, though no major AI lab appears to be taking that approach yet.

AI coding assistants, the tools that help developers write programs faster by suggesting or generating code automatically, are getting remarkably good at one thing: finding and exploiting weaknesses in software. They're not getting nearly as good at the other thing everyone hoped they would do, which is write safe software in the first place.

Several research teams have now measured that gap carefully.

How bad is the problem, exactly?

Bad enough to be a consistent pattern across independent studies. Veracode, an application security company, tested more than 100 versions of AI coding models over the past year and found that 44% of the code those models produced contained at least one flaw from the OWASP Top 10, a widely recognised list of the most common and dangerous software vulnerabilities. Even the best-performing models failed the security check roughly one time in three.

What makes that figure striking is the contrast with syntax, the basic grammar of code. Every model tested got syntax right 99% of the time. The models are nearly perfect at writing code that runs. They're far less reliable at writing code that runs safely.

Dutch software quality firm Software Improvement Group found a similar pattern: AI-generated code produces roughly twice the security-rule violations compared with code written by humans. Its chief technology officer, Jasper Geurts, was careful to note that AI didn't create the underlying problem. It amplified one that already existed.

Why can't AI just fix the holes it finds?

Fixing a flaw turns out to be harder than spotting one. Exploiting a vulnerability requires finding one working path in. Patching it properly means closing every possible path, including ones that look slightly different in the code but lead to the same weakness.

Researchers at 1Password's Off-By-1 Labs saw this directly. They asked several AI models to generate patches for six real, previously fixed vulnerabilities in complex open-source software. Out of more than 6,000 attempts, the AI produced a correct, complete fix only 26% of the time. More than half the AI-generated patches either failed to close the hole, introduced a brand-new flaw, or did both at once. We covered that study on 7 August in "AI Patches Security Flaws Correctly Only 26% of the Time, 1Password Study Finds"; these new findings from Veracode and Xint put it in broader context.

The team noted in section 7.3 of their research paper that an AI agent sometimes fixed only the specific version of a bug shown in a test case, while missing character-for-character identical copies of the same bug sitting in nearby code. That points to a structural limitation in how current AI models process context, not simply a training gap that more data will cure.

What should developers, and their organisations, do right now?

No single fix exists yet. Teams should treat AI-generated code the way they'd treat code from a junior developer who works very fast: review it, run automated security scans against it, and never assume it's safe simply because it compiles and runs.

For ordinary people the implication is indirect but real. Software built partly or entirely by AI tools, which now means a large and growing share of apps and websites, may carry weaknesses that the developers themselves didn't write and may not have noticed. If a service announces a data breach, check whether your login credentials or personal details were exposed, change your password for that service, and watch for unusual activity on any linked accounts.

Veracode's Chris Wysopal, the company's chief security evangelist, told CSO Online he believes the root fix sits with the AI labs themselves. They should stop training on the broad, mixed-quality public code available online and instead license tightly reviewed code from industries where correctness is non-negotiable, such as banking software or aviation systems. As of now, no major lab appears to be doing that.

Common questions

Does this mean AI coding tools are too dangerous to use?

Not necessarily, but they need supervision. AI tools can speed development considerably when teams also run security scans and code reviews on everything the AI produces. The risk comes from treating AI output as finished, verified work.

Will this get better as AI models improve?

Possibly, but not automatically. Researchers point out that the attention mechanisms built into current AI models, the way a model judges the context and relevance of each piece of input, may make complete vulnerability patching structurally difficult regardless of how much additional training a model receives. Better-quality training data aimed specifically at secure coding would help more than simply training on more code.