Overview of CWE-bench v2
Collinear AI has released CWE-bench v2, a public suite consisting of 100 tasks aimed at evaluating advanced cybersecurity capabilities in identifying and rectifying security vulnerabilities. This suite encompasses 91 distinct Common Weakness Enumerations (CWEs), emphasizing the necessity for models to go beyond mere pattern recognition.
Task Design and Verification
Each task provides an agent with access to a real open-source repository along with concise instructions to address a vulnerability while preserving existing functionality. The scoring mechanism is entirely programmatic and deterministic, ensuring that results can be reproduced reliably. Collinear AI has collaborated with LLM judge panels to curate programmatic checks that enhance both the difficulty and fairness of the evaluations. Additionally, every task has undergone manual review to ensure quality.
The project includes a held-out set of tasks that have not been previously encountered by models, allowing for ongoing progress tracking. Teams interested in evaluating their models against this held-out set are encouraged to reach out via cyber@collinear.ai.
Results Tracking and Cost Analysis
Results from the evaluations are monitored in two distinct locations: a leaderboard for open-weight models on the CWE-bench dataset page and a CWE-bench leaderboard Space that incorporates API-only models. Among the 11 models assessed, Gemini 4 Argon leads with a pass rate of 67.8% at pass@1 and 81.0% at pass@4, closely followed by Claude Opus 5.5 with 67.3% at pass@1 and 78.0% at pass@4. Notably, 95 out of the 100 tasks have been successfully solved by at least one model, although the leading model still fails 19% of the tasks on each attempt.
In terms of cost, Collinear AI identifies a Pareto frontier comprising DeepSeek V4.1 Flash (1017), Claude Opus 5.5, and Gemini 4 Argon. DeepSeek achieves a pass rate of 57.0% at a cost of $0.04 per rollout, while Opus improves the pass rate by approximately ten points at $1.83 per rollout, and Gemini further advances at $3.92 per rollout.
Running the Benchmark
All tasks are packaged as Harbor environments, which include Dockerized tasks, instructions, and deterministic verifiers. The dataset also includes a registry.json file, enabling users to run the benchmark directly from the repository. To initiate the benchmark, users must accept the dataset access conditions, save their Hugging Face token to git credentials, and utilize Harbor to fetch and execute the tasks. Example commands provided by the authors include:
harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 --agent --model --env docker --allow-agent-host
hf auth login --token $HF_TOKEN --add-to-git-credentialAccess to the public dataset is gated and contingent upon accepting the access conditions outlined on the dataset page prior to running the benchmark.
Original source: Hugging Face Blog