AI Coding Tools Carry Real Security Risks, and the Danger Depends on What You're Building With
A new study tested 16 major AI coding assistants and found an average of 15 security flaws per project. The safest choice for one type of software can be one of the worst for another.
Key points
- Researchers analysed 1,760 complete software projects generated by 16 AI coding tools and found more than 27,000 security flaws in total.
- The average AI-generated project contained 15 confirmed vulnerabilities, including 4.3 rated critical or high severity.
- Two Anthropic Claude models scored best on security (80.4 and 76.4 out of 100), while OpenAI GPT-4o Mini scored lowest at 21.6.
- The highest-risk software frameworks carried 40 times more exposure than the safest ones, according to software governance firm Secure Code Warrior.
- Seventeen specific weakness categories appeared in every single model tested, with five accounting for the largest share of real-world risk.
AI coding assistants, tools that write software code on a developer's behalf, have spread quickly across companies of every size. They save time and cut costs. According to a new study, they also reliably introduce security weaknesses that companies are only beginning to measure.
Software governance firm Secure Code Warrior published its AI Trust Index this week, built on research conducted with RMIT University. Researchers tested 16 AI models from vendors including OpenAI, Anthropic and Google, generating 1,760 complete software projects and scanning each one for vulnerabilities, meaning flaws that an attacker could use to break into a system or steal data.
Across all projects, researchers identified more than 27,000 vulnerabilities. The average project carried 15 confirmed flaws, with 4.3 of those rated critical or high severity.
Does it matter which AI tool your company uses?
Yes, but the answer is more complicated than a simple ranking suggests.
Models were scored from 0 to 100, where a higher number means fewer security problems. Anthropic's Claude Sonnet 4.5 and Claude Opus 4.5 led the field with scores of 80.4 and 76.4 respectively. OpenAI's GPT-4o Mini came last at 21.6. That sounds like a clear result. It isn't.
A model's score shifts dramatically depending on which software framework, meaning the underlying toolkit a developer uses to build an application, is being tested. Claude Opus 4.5 scored a perfect 100 building applications in Django, a popular web framework, then dropped to 57.3 in one C programming environment and just 28.5 in another. Eleven frameworks were tested in total, and the highest-risk ones carried forty times more exposure than the safest. Our earlier report on how GitHub Copilot produces harmful code through indirect prompting showed a different dimension of the same problem: AI tools misbehave in ways their aggregate scores don't capture.
JavaScript and Java EE carried most of the danger. C# and Java Spring barely registered.
Cost offered no protection. Researchers found little correlation between what a company pays for a model and how safely it writes code.
The failures weren't random, either. Each model produced the same categories of mistakes repeatedly. The most common problems were things the AI forgot to do: checking that a user is who they claim to be (authentication) and checking that data entered into a form is safe before processing it (input validation). Seventeen weakness categories appeared across all 16 models tested.
Secure Code Warrior CEO Pieter Danhieux highlighted five that carry the most volume: sensitive data leaking into log files, cross-site scripting (where attackers inject malicious code into web pages that other users then load), hard-coded credentials (passwords written directly into the software itself), predictable session tokens (the digital passes that keep a user logged in), and path traversal (a technique that tricks software into accessing files it shouldn't). "None of these is a new, insurmountable threat vector," Danhieux told Dark Reading. "AppSec teams have had playbooks for all five for years. The job now is making sure those playbooks get pointed at AI-generated code with the same rigor as anything else."
Should you worry about your team's specific setup?
The framework matters more than the model name on the licence. That's the finding worth sitting with. Your security team can't audit every framework choice, but it can require that any team building in a high-risk environment clear a security gate before shipping, the same discipline applied to any other elevated-risk system. Match scrutiny to the framework, not just to the AI tool that generated the code.
