- Home
- MLOps Platforms
- MLflow GenAI Evaluation
MLflow GenAI Evaluation
What it is
MLflow GenAI Evaluation helps teams measure, improve, and monitor LLM applications and AI agents from development through production. Evaluation Datasets centralize test cases, expected answers, and evaluation data. Teams can collect feedback from end users and domain experts, attach it to traces, and keep metadata with that feedback. Scoring includes LLM-as-a-Judge, custom LLM judges, and code-based scorers; automatic evaluation supports LLM judges but not code-based scorers. The MLflow UI presents per-record results and comparisons between agent versions. MLflow Tracing records metrics such as latency and token usage for production monitoring and lists integrations with more than 40 LLM and agent libraries, including OpenTelemetry, LangChain, OpenAI, and Anthropic. Built-in adapters support judge providers including Anthropic, Bedrock, Google, and xAI. The open-source evaluation API and UI are free, with API, self-hosted, and web options. Optional basic HTTP authentication controls experiments, registered models, and scorers, requiring a configured CSRF secret key and an admin password when enabled.
For and against
For
- Centralized evaluation datasets
- Supports LLM and code-based scorers
- Compares results across agent versions
- Tracing records latency and token usage
- Free open-source API and UI
Against
- Automatic evaluation excludes code-based scorers
- HTTP authentication requires setup credentials
- Designed for LLM applications and agents
The verdict
MLflow GenAI Evaluation combines dataset management, human feedback, automated judging, and production tracing. Teams can deploy it through API, self-hosting, or the web without a listed price.
Who it suits
It is designed for teams building and operating LLM applications or AI agents that need evaluation, feedback, tracing, and regression comparisons.
Plans and prices
MLflow GenAI Evaluation
Open source; evaluation API and UI
Read 29 Sep 2026 · mlflow.org
On the record
- Access control detail
- Basic HTTP authentication requires a configured secret key for CSRF protection and an admin password when first enabled.mlflow.org · read 29 Sep 2026
- Automatic evaluation limit
- Automatic evaluation supports LLM judges; code-based scorers are not supported in that mode.mlflow.org · read 29 Sep 2026
- Evaluation data
- Evaluation Datasets provide a centralized place to manage test cases, ground-truth expectations, and evaluation data.mlflow.org · read 29 Sep 2026
- Human feedback
- Users can collect and manage feedback from end users and domain experts, with feedback attached to traces and recorded with metadata.mlflow.org · read 29 Sep 2026
- Integrations
- MLflow Tracing lists integrations with 40+ LLM and agent libraries and frameworks, including OpenTelemetry, LangChain, OpenAI, and Anthropic.mlflow.org · read 29 Sep 2026
- Judge providers
- MLflow's evaluation quickstart says judge models can use providers including Anthropic, Bedrock, Google, and xAI through built-in adapters.mlflow.org · read 29 Sep 2026
- Judges and scorers
- MLflow includes LLM-as-a-Judge scorers and supports custom LLM judges and code-based scorers.mlflow.org · read 29 Sep 2026
- Production monitoring
- MLflow Tracing captures metrics such as latency and token usage to help monitor application performance in production.mlflow.org · read 29 Sep 2026
- Purpose
- MLflow helps teams measure, improve, and monitor the quality of LLM applications and AI agents from development through production.mlflow.org · read 29 Sep 2026
- Review and compare
- Evaluation results can be reviewed in the MLflow UI, including per-record results and comparisons across agent versions.mlflow.org · read 29 Sep 2026
- Security controls
- MLflow supports optional basic HTTP authentication for access control over experiments, registered models, and scorers.mlflow.org · read 29 Sep 2026
- Who it is for
- The product is designed for teams building and operating LLM applications and AI agents who want evaluation and monitoring across the application lifecycle.mlflow.org · read 29 Sep 2026
