Anthropic released their latest frontier model Opus 5, and they report its security capabilities as follows:
[Opus 5] comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats.
This is illustrated by Opus 5's performance on OSS-Fuzz, an evaluation we've developed to assess how well models can find and then exploit vulnerabilities without extensive human guidance.
Anthropic, and other labs, tend to frame their models' security capabilities solely in terms of how the model reacts to existing security threats, by "finding" and "exploiting" vulnerabilities. Both of these capabilities only become relevant once a vulnerability has already been written into a codebase, which means the entire security evaluation is scoped to the period after the mistake has been made.
But this is only half of the picture and, as we at Corridor argue, it's becoming the less important half. We believe that in order to loosen the ever-tightening bottleneck of code review, we must shift security left by proactively preventing insecure code from being generated in the first place.
This is why it's important for model providers to report not just reactive security capabilities, but proactive security capabilities.
What Are Proactive Security Capabilities?
Proactive security capabilities are a model's ability to avoid introducing security risk into the code it writes. This includes the model's ability to generate code free of security vulnerabilities, and its ability to produce code that aligns with a user's security policy.
A key part of producing secure code is being able to retrieve relevant security context when presented with a plan of a feature to be implemented in a codebase. If the model can retrieve this security context, it can then align generated code with it, producing overall more secure code. We call this retrieval task: security context retrieval.
To measure a model's ability to perform security context retrieval, we built a novel benchmark we call PremortemBench and ran Opus 5 and other frontier models against it.
Results
As expected, Opus 5 posts the highest F1 of any model we tested, making it the strongest frontier model available today on this task. However, its F1 is only 0.632, and that the six models behind it all fall between 0.570 and 0.607. In other words, no frontier model is particularly good at security context retrieval, and the differences between them on this task are small. Even the latest Anthropic model only retrieves about three quarters of the security context that applies to a given change, meaning it can't adequately steer an agent to produce secure code for about a quarter of code changes.
Reactive vs. Proactive: A Tale of Two Capabilities
Now compare these numbers to Opus' performance on a reactive task, like Anthropic's OSS-Fuzz benchmark, which measures a model's ability to discover and exploit known vulnerabilities. Opus 5 successfully identifies vulnerabilities in 79.4% of OSS-Fuzz targets, compared to 61.5% for Opus 4.8, garnering a 29% improvement. However, on our proactive security evaluation, Opus 5's only moved up 5% over its predecessor.
While Opus got meaningfully better at finding vulnerabilities, it barely improved at avoiding them. This tells us that reactive and proactive security capabilities are distinct, and that improvement in one does not give us improvement on the other for free. Thus in order to really measure models' performance on cybersecurity tasks, we must also evaluate them on proactive security tasks like PremortemBench.
We'll be releasing more about PremortemBench soon, but in the meantime, check out our talk on proactive security evaluations on Friday, August 7 at 4pm at DEF CON!