- Home
- LLM Evaluation Tools
- OpenCompass
OpenCompass
What it is
OpenCompass is a free, open-source Python package for self-hosted evaluation of large language and large vision-language models. It assesses models across datasets covering knowledge, reasoning, coding, science, language, long context, and safety, with more than 100 datasets described. Supported providers include OpenAI, Anthropic, Gemini, Qwen, GLM, and DeepSeek, alongside open-source models. Evaluations can use zero-shot, few-shot, or chain-of-thought methods with standard or dialogue prompts. The platform supports task division and distributed, data-parallel evaluation across multiple GPUs. Configuration files record experiments, and results can be reported in real time. Its modular design allows extensions for models, datasets, task strategies, and cluster systems. LMDeploy and vLLM are listed inference backends, while VLMEvalKit adds multimodal loading, OpenAI-compatible inference, and its metrics. The project uses an Apache-2.0 license.
For and against
For
- Supports more than 100 datasets
- Evaluates API and open-source models
- Offers zero-shot and few-shot methods
- Runs distributed evaluation across GPUs
- Includes multimodal evaluation integration
Against
- Some datasets need extra dependencies
- Agent evaluations may cause dependency conflicts
- Python requirements differ by dataset code
- Self-hosted setup required
The verdict
OpenCompass provides a broad, extensible framework for model benchmarking across text and multimodal tasks. Its dependency caveats and Python-version guidance should be reviewed before setting up evaluations.
Who it suits
It is aimed at researchers and practitioners evaluating large language and vision-language models. Self-hosted teams needing custom metrics, safety evaluations, or distributed runs may find its modular design relevant.
Plans and prices
OpenCompass
Open-source software · Python package · Some datasets and inference backends require optional dependencies
Read 7 Oct 2026 · github.com
On the record
- Community resources
- The project links to CompassHub for benchmark browsing, CompassRank for leaderboards, documentation, Discord, and WeChat.github.com · read 7 Oct 2026
- Dataset caveat
- The documentation says some datasets need extra dependencies, and some agent evaluations may require many dependencies that can conflict with existing environments.opencompass.readthedocs.io · read 7 Oct 2026
- Distributed evaluation
- It supports task division and distributed evaluation, including data-parallel evaluation across multiple GPUs.github.com · read 7 Oct 2026
- Evaluation methods
- It supports zero-shot, few-shot, and chain-of-thought evaluations with standard or dialogue prompt templates.github.com · read 7 Oct 2026
- Experiment tracking
- Configuration files record experiments, and the project supports real-time reporting of results.github.com · read 7 Oct 2026
- Extensibility
- Users can extend the modular system with models, datasets, task division strategies, and cluster management systems.github.com · read 7 Oct 2026
- Inference integrations
- The installation documentation lists LMDeploy and vLLM as inference backends and provides an API installation option.opencompass.readthedocs.io · read 7 Oct 2026
- Installation requirements
- The README recommends Python 3.12, while users relying on pyext-backed code evaluation datasets are instructed to use Python 3.10.github.com · read 7 Oct 2026
- Intended users
- The project describes CompassHub and CompassKit as tools for researchers and practitioners evaluating large language and large vision-language models.github.com · read 7 Oct 2026
- License
- The GitHub repository identifies its license as Apache-2.0.github.com · read 7 Oct 2026
- Model APIs
- OpenCompass supports evaluating API models alongside open-source models and lists API integrations including OpenAI, Gemini, Claude, Qwen, and others.github.com · read 7 Oct 2026
- Models and datasets
- The project describes support for models from providers including OpenAI, Anthropic, Gemini, Qwen, GLM, and DeepSeek, and more than 100 datasets.github.com · read 7 Oct 2026
- Multimodal support
- The project says its VLMEvalKit integration enables multimodal dataset loading, inference through OpenAI-compatible APIs, and evaluation with VLMEvalKit metrics.github.com · read 7 Oct 2026
- Purpose
- OpenCompass is an LLM evaluation platform for assessing models across datasets covering knowledge, reasoning, coding, science, language, long context, and safety.github.com · read 7 Oct 2026
