cyber-jev / results
Measured in review decisions.
Offline evaluation on C/C++ functions. Reported Cyber-JEV results use the five-model ensemble unless stated otherwise.
1Overall ranking
How well each system ranks vulnerable functions above the rest, across 7,270 test functions.
AUROC+0.071
Average precision+0.096
Length-stratified AUROC+0.062
Reference-only fine-tuning reaches 0.804 AUROC. The small aggregate difference from Cyber-JEV is not evidence of a conclusive AUROC advantage over that baseline; patch-pair behavior provides a separate comparison. These are results for the reported benchmark, not a universal ranking of tools.
2Fixed review budget
Both systems flag exactly 4,031 functions. How many of the 322 vulnerable functions does each one find?
39 missed
16 missed
Vulnerable functions found+23
False alarms−23
The 4,031-alert budget is the number of functions Jev 1.13 flags at its default threshold. Cyber-JEV still misses 16 vulnerable functions.
3Patch sensitivity
The share of vulnerable–patched pairs in which the vulnerable version receives the higher score. Ties count as half. The shaded length of each row shows the percentage.
−Reference data only+Cyber-JEV•Patched-only groups (3 models)
T271 main benchmark+14.8 pts
A72 TitanVul+29.2 pts
P309 TitanVul+14.9 pts
A72 and P309 are external TitanVul suites. Training on patched-only groups scores higher on these pair tests, illustrating the trade-off between global ranking and patch sensitivity.
Evaluation separates protected splits from training by project, CVE, fix commit and code similarity. The manuscript reports project-cluster bootstrap intervals and patch-pair tests. Dataset labels and commit-derived supervision retain their original limitations.
A role in the review workflow
A first pass.
A human decision.
The intended use is to rank changed functions before deeper review with static analysis, rules or a security reviewer.
The work evaluates historical functions and fixes. Live pull-request deployment, new vulnerability discovery and comparative throughput have not been demonstrated. A score prioritizes attention; it does not establish that code is safe.