- Home
- LLM Evaluation Tools
- LM Evaluation Harness
LM Evaluation Harness
What it is
LM Evaluation Harness is a free, MIT-licensed framework for evaluating generative language models across standardized and custom tasks. The project lists more than 60 academic LLM benchmarks, covering hundreds of subtasks and variants. It supports Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, vLLM, hosted OpenAI and Anthropic APIs, and local servers compatible with OpenAI APIs. Users can define prompts and metrics, configure tasks with YAML, and extend model backends, filters, metrics, or aggregations through external packages or local modules. Publicly available prompts are intended to support reproducibility and comparison between papers. The core package is installed separately from model backends and task dependencies, which are optional extras. Documented runtime choices include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, and ROCm accelerators. For APIs without logits or log probabilities, documentation limits use to generate-until tasks. Support is available through GitHub issues or EleutherAI Discord.
For and against
For
- More than 60 standard academic benchmarks
- Supports major local models and hosted APIs
- Custom prompts, metrics, and YAML tasks
- Extensible through packages or local modules
- Supports CPU and multiple accelerator providers
Against
- Model backends require separate optional installs
- Task dependencies may need additional installation
- Some APIs only support generate-until tasks
- MIT license provides no warranty
The verdict
LM Evaluation Harness provides a broad, extensible basis for language-model evaluation, from published benchmarks to custom tasks. Installation is modular, and API limitations matter when logits or log probabilities are unavailable.
Who it suits
It suits researchers, organizations, and developers evaluating language models with shared benchmarks or custom metrics. Self-hosted users who can install optional backends and dependencies will have the broadest options.
Plans and prices
LM Evaluation Harness
MIT-licensed software · model backends and task dependencies may require separate optional installs
Read 4 Oct 2026 · github.com
On the record
- API integrations
- The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs.github.com · read 4 Oct 2026
- Benchmarks
- The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants.github.com · read 4 Oct 2026
- Custom evaluation
- Users can define custom prompts and evaluation metrics, and can configure tasks with YAML.github.com · read 4 Oct 2026
- Installation
- The base package provides the core framework while model backends are installed separately as optional extras.github.com · read 4 Oct 2026
- Model support
- Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM.github.com · read 4 Oct 2026
- Notable limitation
- For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks.github.com · read 4 Oct 2026
- Plugins
- Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module.github.com · read 4 Oct 2026
- Purpose
- LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.github.com · read 4 Oct 2026
- Reproducibility
- The project says its use of publicly available prompts supports reproducibility and comparability between papers.github.com · read 4 Oct 2026
- Security and license
- The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty.github.com · read 4 Oct 2026
- Support
- The project directs users to open a GitHub issue or join the EleutherAI Discord for support.github.com · read 4 Oct 2026
- Supported platforms
- Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators.github.com · read 4 Oct 2026
- Users
- The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations.github.com · read 4 Oct 2026
