Secure Code Warrior launches AI trust index on code risk

Secure Code Warrior has launched the SCW AI Trust Index, a benchmark that measures security risks in code generated by artificial intelligence models.

The index is based on an assessment of 1,760 complete codebases generated by 16 large language models from groups including OpenAI, Anthropic, Google and Alibaba. The testing found that AI-generated code contained an average of 15 confirmed vulnerabilities per codebase, including 4.3 classified as severe.

The methodology was developed with RMIT University in Australia and later extended internally. Secure Code Warrior describes the index as a live benchmark that expands as new models are added, rather than a single research snapshot.

The findings indicate that the security weaknesses were neither evenly distributed nor random. The study identified 86 unique Common Weakness Enumerations across the codebases, with recurring problems centred on logging failures, injection flaws, insecure design and broken access control.

The most common weakness was CWE-532, or insertion of sensitive information into log files, which appeared 8,543 times as a true positive in the results. That pattern suggests AI coding tools tend to reproduce the same kinds of security errors repeatedly.

Model patterns

The research argues that each model showed a distinct security profile across the frameworks tested. Rather than one model emerging as the safest in all circumstances, the rankings shifted depending on the software framework used.

In the tests, GPT-5.1 performed best in Java Enterprise API, Claude Sonnet 4.5 led in Java Spring, Claude Opus 4.8 ranked highest in Python Django, GPT-5.5 topped C# .NET, and Claude Fable 5 led in C. The results suggest that security outcomes depend on the interaction between the model and the development environment.

That matters for companies introducing AI coding assistants into software teams, because model choice may need to vary by project rather than follow a single company-wide standard. The findings also indicate that security leaders may need more detailed oversight of how developers use these systems in specific workflows.

Pieter Danhieux outlined the company's position on the results.

"Every AI model we tested leaves a predictable, repeatable pattern of security gaps and weaknesses. Developers are also predictable in that they aren't going to abandon their preferred model over a security score. The SCW AI Trust Index was purpose-built to help CISOs and security leaders manage the models already in use; there is no identified "winner", but this data provides the crucial insights needed to truly manage AI tools safely, with consideration to those inherent security gaps, and allow the right guardrails and developer learning pathways to be brought to life in a modernised security program. AI-generated code needs the same scrutiny we've always given human-written code, and now we finally have the data to know exactly where to look," said Pieter Danhieux, Co-Founder & Chief Executive Officer, Secure Code Warrior.

Cost and risk

The study also found no clear link between the price of using a model through an application programming interface and the security quality of the code it produced. GPT-5.1 recorded one of the strongest trust index scores at USD $5.60 per run, while some more expensive models produced weaker outcomes.

That finding challenges the assumption that a higher model cost brings lower coding risk. For procurement and security teams, it suggests that price and security should be assessed separately when selecting AI tools for developers.

The index is designed to give organisations a way to compare models using security data, identify recurring vulnerability patterns and quantify risks from AI-generated code. Its launch comes as more software teams hand routine coding tasks to generative AI systems, increasing pressure on security teams to understand the weaknesses those tools may introduce.

Secure Code Warrior says the benchmark is intended to support governance around AI-assisted development rather than declare one model universally superior. Its results instead point to a more fragmented picture, with risk varying by model, framework and the combination of both.

Across all the codebases tested, the central conclusion was that AI coding risk can be measured and anticipated rather than treated as an unknowable by-product of automation. The study identified 86 unique CWEs, with the most common categories clustering around logging failures, injection vulnerabilities, insecure design and broken access control.