- Home
- LLM Security Tools
- CyberSecEval
CyberSecEval
What it is
CyberSecEval 4 is a benchmark suite for assessing cybersecurity weaknesses and defensive capabilities in large language models. Its MITRE tests examine responses to cyberattack requests, while False Refusal Rate tests measure incorrect refusals of borderline benign queries. Secure-code checks look for insecure suggestions in instruction and autocomplete settings. Prompt injection evaluations include textual, multilingual, and visual attacks, with a judge LLM assessing whether injected instructions succeeded. A code-interpreter benchmark evaluates compliance with malicious prompts and groups judged responses into three categories: extremely malicious, potentially malicious, and non-malicious. AutoPatchBench measures patch-generation agents on C and C++ vulnerabilities found by fuzzing. CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning for defensive use. The suite supports API, Linux, and self-hosted setups. Its guide lists API support for OpenAI, Anyscale, and Together, and describes adding custom providers for self-hosted models. Setup uses a Python virtual environment, requirements installation, and a command-line module. The benchmarks are licensed under MIT for research and commercial use.
For and against
For
- Tests cover prompt injection and insecure code suggestions.
- Includes benchmarks for patch-generation agents.
- CyberSOCEval covers defensive malware and threat analysis.
- Supports API, Linux, and self-hosted use.
- MIT license permits research and commercial use.
Against
- AutoPatch requires Podman and substantial computing resources.
- AutoPatch recommends at least 80 CPUs and 8 TB storage.
- Apple Silicon MacBooks are not currently supported for AutoPatch.
The verdict
CyberSecEval covers several distinct cybersecurity behaviors, from prompt injection to defensive threat reasoning. AutoPatchBench has substantial resource requirements and does not currently support Apple Silicon MacBooks.
Who it suits
CyberSecEval suits people evaluating cybersecurity risks or defensive behavior in large language models. Its varied benchmark areas and provider options may fit teams working with API-based or self-hosted models.
Plans and prices
CyberSecEval 4
Open source benchmark suite
Read 8 Oct 2026 · github.com
On the record
- Automated evaluation
- The original research describes automated test-case generation and an evaluation pipeline for LLM cybersecurity safety.arxiv.org · read 15 Oct 2026
- AutoPatchBench
- AutoPatchBench measures AI patch-generation agents on C/C++ vulnerabilities found by fuzzing.meta-llama.github.io · read 8 Oct 2026
- AutoPatch requirements
- AutoPatch requires Podman and substantial compute and storage; the page recommends at least 80 CPUs and 8 TB storage and says Apple Silicon MacBooks are not currently supported.meta-llama.github.io · read 8 Oct 2026
- Code interpreter
- The code interpreter benchmark assesses whether models comply with malicious prompts and categorizes judged responses as extremely malicious, potentially malicious, or non-malicious.meta-llama.github.io · read 8 Oct 2026
- Company founded
- Meta’s SEC filing says the company was incorporated in Delaware in July 2004.sec.gov · read 15 Oct 2026
- Company headquarters
- Meta lists its principal executive offices at 1 Meta Way, Menlo Park, California 94025.sec.gov · read 15 Oct 2026
- Company legal name
- Meta’s SEC filing identifies the maker as Meta Platforms, Inc., a Delaware corporation.sec.gov · read 15 Oct 2026
- Company size
- LinkedIn lists Meta as having more than 10,001 employees and headquarters in Menlo Park, California.linkedin.com · read 15 Oct 2026
- Current version
- CyberSecEval 4 adds AutoPatchBench for measuring an LLM agent’s ability to automatically patch vulnerabilities in native code.meta-llama.github.io · read 15 Oct 2026
- CyberSecEval 2 additions
- CyberSecEval 2 added prompt-injection, code-interpreter-abuse and vulnerability-exploitation evaluations.arxiv.org · read 15 Oct 2026
- CyberSecEval 3 coverage
- CyberSecEval 3 evaluates eight risks across risks to third parties and risks to application developers or end users.arxiv.org · read 15 Oct 2026
- CyberSecEval 3 features
- CyberSecEval 3 added automated social engineering, scaled manual offensive operations and autonomous offensive operations tests.arxiv.org · read 15 Oct 2026
- Documentation depth
- The repository documents benchmark commands, provider formats, self-hosted integration, datasets, judging stages and developer tests.github.com · read 15 Oct 2026
- Extensibility
- Developers can add providers by implementing an LLM class, registering it in the factory and supplying query and chat methods.github.com · read 15 Oct 2026
- False-refusal metric
- CyberSecEval 2 introduced False Refusal Rate to quantify the tradeoff between rejecting unsafe requests and answering benign ones.arxiv.org · read 15 Oct 2026
- Independent assessment
- The peer-reviewed critique describes CyberSecEval as useful in a young field but identifies features that limit its utility.arxiv.org · read 15 Oct 2026
- Independent cost observation
- An Inspect Evals report estimated full one-epoch CyberSecEval 4 runs at roughly $29 to $213 per model depending on the model used.github.com · read 15 Oct 2026
- Independent implementation limit
- Inspect Evals intentionally omits the autonomous-uplift and autopatching prototypes from its public CyberSecEval 4 task surface.github.com · read 15 Oct 2026
