- Home
- AI LLM Evaluation Tools
- Pydantic Evals
On the recordRead 30 Sep 202618 facts
Pydantic Evals
On the record
- Agent behavior checks
- Built-in evaluator examples cover tool correctness, trajectory matching, argument correctness, maximum tool calls, and maximum model requests.pydantic.dev · read 29 Sep 2026
- Code-first workflow
- Evaluation components are defined in Python, and experiment reports can be printed, serialized, or viewed in Pydantic Logfire.pydantic.dev · read 30 Sep 2026
- Cost consideration
- LLM-as-a-judge evaluations can take seconds, cost money, and produce non-deterministic results; deterministic evaluators are described as having no cost.pydantic.dev · read 29 Sep 2026
- Costs and tradeoffs
- LLM judges can be slower, cost money, be non-deterministic, and have biases.pydantic.dev · read 30 Sep 2026
- Datasets and cases
- Datasets group test cases, which can include inputs, expected outputs, metadata, and case-specific evaluators.pydantic.dev · read 30 Sep 2026
- Dependencies
- Pydantic Evals does not depend on pydantic-ai, and Logfire is optional.pydantic.dev · read 29 Sep 2026
- Evaluators
- It includes built-in deterministic evaluators and supports custom evaluators, LLM judges, and evaluators that return assertions, scores, or labels.pydantic.dev · read 29 Sep 2026
- Evaluator types
- Evaluators include deterministic checks, LLM judges, span-based and agentic evaluators, and custom evaluators.pydantic.dev · read 30 Sep 2026
- Installation
- The package installs with `pip install pydantic-evals` or `uv add pydantic-evals`.pydantic.dev · read 30 Sep 2026
- Logfire integration
- Logfire is an optional dependency for using OpenTelemetry traces in evaluations or sending evaluation results to Logfire.pydantic.dev · read 30 Sep 2026
- Maker
- Pydantic describes itself as a remote-first company.pydantic.dev · read 30 Sep 2026
- Online evaluation
- Evaluators can run in the background on every production or staging call, or on a sampled subset of traffic.pydantic.dev · read 30 Sep 2026
- Production evaluation
- Online evaluation can attach evaluators to production or staging traffic so every call or a sampled subset is graded in the background.pydantic.dev · read 29 Sep 2026
- Purpose
- Pydantic Evals tests and evaluates AI systems, from simple LLM calls to complex multi-agent applications.pydantic.dev · read 30 Sep 2026
- Related service security
- Pydantic says Logfire is SOC 2 Type 2 audited, GDPR aligned, and can support HIPAA workloads under a signed Business Associate Agreement.pydantic.dev · read 30 Sep 2026
- Security
- Pydantic’s Logfire security page states that it has SOC 2 Type 2 audited controls, GDPR compliance, and HIPAA support under a signed Business Associate Agreement.pydantic.dev · read 29 Sep 2026
- Third-party integrations
- The documentation shows adapters for Ragas and DeepEval, which are optional dependencies and are not installed with Pydantic Evals.pydantic.dev · read 30 Sep 2026
- What it evaluates
- It can grade an agent’s final outputs and tool-call trajectory against datasets or sampled live production traffic.pydantic.dev · read 30 Sep 2026
