- Home
- LLM Evaluation Tools
- SWE-bench
On the recordRead 9 Oct 202619 facts
SWE-bench
On the record
- Benchmark variants
- The site lists SWE-bench Lite, Verified, Multimodal, and Multilingual datasets alongside the original benchmark.swebench.com · read 8 Oct 2026
- CLI
- SWE-bench CLI can submit model predictions for evaluation, retrieve reports, manage evaluation runs, and track performance in the cloud.swebench.com · read 8 Oct 2026
- Cloud evaluation
- Evaluations can run in the cloud using Modal, or be submitted through the sb-cli tool.swebench.com · read 9 Oct 2026
- Cloud evaluations
- The documentation describes cloud-based evaluations using Modal and the sb-cli tool.swebench.com · read 8 Oct 2026
- Dataset variants
- The project lists SWE-bench, SWE-bench Lite, SWE-bench Verified, and SWE-bench Multimodal as available datasets.swebench.com · read 9 Oct 2026
- Evaluation limit
- Docker is required for consistent evaluation environments, according to the FAQ.swebench.com · read 8 Oct 2026
- Evaluation limits
- The guide notes that caching all Docker images can require about 2,000 GB of storage.swebench.com · read 9 Oct 2026
- Evaluation method
- Evaluation applies generated patches to repositories and runs tests in Docker containers to check whether issues are resolved.swebench.com · read 9 Oct 2026
- Installation
- The project requires Python 3.9 or later and Docker for the evaluation environment.swebench.com · read 9 Oct 2026
- Integrations
- The project documents using Hugging Face datasets to load SWE-bench data and Modal for cloud evaluations.swebench.com · read 9 Oct 2026
- Multilingual dataset
- SWE-bench Multilingual contains 300 tasks from 42 repositories across nine programming languages.swebench.com · read 9 Oct 2026
- Multimodal dataset
- SWE-bench Multimodal contains 480 instances with issues described using visual elements.swebench.com · read 9 Oct 2026
- Operating systems
- The Docker setup guide provides installation instructions for macOS, Linux, and Windows, with WSL 2 required on Windows.swebench.com · read 9 Oct 2026
- Original dataset
- The original benchmark contains 2,294 task instances drawn from pull requests and issues in 12 Python repositories.swebench.com · read 9 Oct 2026
- Purpose
- SWE-bench evaluates language models on real-world GitHub software issues by asking them to generate patches that resolve the issues.swebench.com · read 9 Oct 2026
- Resource requirements
- The Docker guide recommends at least 120 GB of free disk space and 16 GB or more of RAM.swebench.com · read 9 Oct 2026
- Support
- The contact page directs general inquiries to the SWE-bench team by email.swebench.com · read 9 Oct 2026
- Verified dataset
- SWE-bench Verified is a human-filtered subset containing 500 instances.swebench.com · read 9 Oct 2026
