- Home
- LLM Evaluation Tools
- LiveBench
On the recordRead 9 Oct 202618 facts
LiveBench
On the record
- Agentic coding requirements
- Evaluating agentic coding tasks requires Docker, and the README says task-specific images may require up to 150 GB of storage.github.com · read 9 Oct 2026
- Benchmark scope
- The current site describes 23 objective tasks across 7 categories, refreshed every six months.livebench.ai · read 4 Oct 2026
- Categories
- The website lists Reasoning, Coding, Agentic Coding, Mathematics, Data Analysis, Language, and Instruction Following.livebench.ai · read 9 Oct 2026
- Contamination controls
- The repository README says questions draw on recently released datasets, papers, news articles, and IMDb movie synopses to limit potential contamination.github.com · read 9 Oct 2026
- Cost metric
- The site defines cost per successful task as (Σ cost ÷ Σ questions ÷ score) × 100 over the selected scope.livebench.ai · read 4 Oct 2026
- Current benchmark
- The website describes 23 objective tasks across 7 categories, refreshed every six months.livebench.ai · read 9 Oct 2026
- Insights
- The website provides quality-versus-cost, cost-ranked, and category-profile views.livebench.ai · read 9 Oct 2026
- Leaderboard
- The leaderboard lets visitors sort overall and category scores, inspect subtask scores, and compare cost per successful task.livebench.ai · read 9 Oct 2026
- Local inference
- The README describes local model inference as unmaintained and recommends serving models through an OpenAI-compatible API using vLLM.github.com · read 9 Oct 2026
- Local models
- The README says local model inference is unmaintained and recommends serving models through an OpenAI-compatible API using vLLM.github.com · read 4 Oct 2026
- Model compatibility
- The README says OpenAI-compatible API endpoints can be used and lists implemented inference support for Anthropic, Cohere, Mistral, Together, and Google models.github.com · read 4 Oct 2026
- Model endpoints
- The repository documents custom OpenAI-compatible API endpoints and inference support for Anthropic, Cohere, Mistral, Together, and Google models.github.com · read 9 Oct 2026
- Open source
- The project publishes its code in a public GitHub repository and links to benchmark data on Hugging Face.github.com · read 4 Oct 2026
- Product
- LiveBench is described as a challenging, contamination-free LLM benchmark.livebench.ai · read 4 Oct 2026
- Purpose
- LiveBench is an LLM benchmark designed to limit test set contamination and use objective evaluation.github.com · read 9 Oct 2026
- Run evaluations
- The repository provides a Python script that can generate model answers, score them, and show results; it supports selecting a model and benchmark subset.github.com · read 9 Oct 2026
- Scoring
- The repository README says questions have verifiable objective ground-truth answers and are scored automatically without an LLM judge.github.com · read 9 Oct 2026
- Support
- The repository invites model evaluation requests by GitHub issue or email at [email protected].github.com · read 9 Oct 2026
