- Home
- AI LLM Evaluation Tools
- OpenAI Evals
On the recordRead 1 Oct 202624 facts
OpenAI Evals
On the record
- Alternative
- OpenAI recommends trying Datasets for a more iterative environment to experiment while building an eval.developers.openai.com · read 30 Sep 2026
- API access
- Evals can be created and managed programmatically through the Evals API or configured in the dashboard.developers.openai.com · read 1 Oct 2026
- Asynchronous scale
- Evals run asynchronously, support larger data volumes, and let users monitor performance across versions.developers.openai.com · read 1 Oct 2026
- Custom endpoint security
- Custom model endpoints must use HTTPS, and OpenAI says it encrypts API keys supplied for them.developers.openai.com · read 30 Sep 2026
- Dashboard
- Evals can be configured and run directly in the OpenAI Dashboard.github.com · read 1 Oct 2026
- Dashboard and API
- You can configure evals in the OpenAI dashboard or programmatically with the Evals API.developers.openai.com · read 30 Sep 2026
- Data handling
- Calls to external models pass data to third parties and are subject to different terms and weaker safety guarantees than calls to OpenAI models.developers.openai.com · read 30 Sep 2026
- Data residency
- The /v1/evals endpoint supports United States and Europe (EEA plus Switzerland) data residency, with service-level support listed.developers.openai.com · read 1 Oct 2026
- Deprecation
- OpenAI says the Evals platform becomes read-only for existing users on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026.developers.openai.com · read 1 Oct 2026
- Evaluation criteria
- An eval uses a data-source schema and testing criteria consisting of graders that determine whether model output is correct.developers.openai.com · read 1 Oct 2026
- External model limits
- Tool calls are currently not supported when running evals with external models.developers.openai.com · read 30 Sep 2026
- External models
- The OpenAI Platform can run evals against third-party models and custom model endpoints.developers.openai.com · read 1 Oct 2026
- Graders
- Supported grader types include string check, text similarity, score model, label model, and Python code execution graders.developers.openai.com · read 1 Oct 2026
- Integrations
- The repository supports logging eval results to Snowflake and says evals can be run and created using Weights & Biases.github.com · read 1 Oct 2026
- Local installation
- The open-source package can be installed locally with pip, and the repository requires Python 3.9 or newer.github.com · read 1 Oct 2026
- Model support
- Evaluations can use OpenAI models, third-party models, and custom model endpoints.developers.openai.com · read 30 Sep 2026
- Private evaluations
- Users can build private evals from their own data without exposing that data publicly.github.com · read 1 Oct 2026
- Purpose
- Evals is a framework for evaluating large language models and LLM systems, with an open-source benchmark registry.github.com · read 1 Oct 2026
