- Home
- AI Agent Evaluation Tools
- Strands Evals
Strands Evals
What it is
Strands Evals is an open-source Python SDK and command-line tool for evaluating agent behavior during development and after deployment. It scores outputs and trajectories, diagnoses failures, probes unsafe behavior, and can simulate users and tools. Evaluators can assess a single output, tool call, trace, or full session, with multiple evaluators combined in an experiment. Built-in groups cover quality, safety, multimodal responses, agent behavior, and skill selection and instruction following. Code-based checks such as Equals, Contains, StartsWith, ToolCalled, StateEquals, and SkillInvoked do not require an LLM judge; developers can also extend the base Evaluator class for domain-specific logic. Install the package with pip as strands-agents-evals to use its Python API or strands-evals CLI. The CLI supports experiment generation, JSON validation, evaluation runs, report rendering, and session diagnosis, with documented CI steps. Trace providers include AWS CloudWatch Logs, Langfuse, and OpenSearch. Evaluations can run against staging or production traces without rerunning agents. The default judge uses Amazon Bedrock with Claude, and the example requires credentials authorized to invoke Claude. Red-teaming is marked experimental.
For and against
For
- Evaluates outputs, tool calls, traces, and full sessions.
- Deterministic checks can run without an LLM judge.
- CLI supports experiment validation, runs, and reports.
- Can evaluate traces without rerunning their agents.
Against
- The default judge requires authorized Claude invocation credentials.
- Some trace integrations require optional package extras.
- The red-teaming API is experimental and may change.
The verdict
Strands Evals offers both code-based checks and judge-driven evaluation, with SDK and CLI workflows spanning development and deployed traces. Plan for required provider access and credentials, and treat red-teaming as experimental.
Who it suits
Strands Evals suits developers who want to validate agent behavior, measure changes, and evaluate agents during development cycles. It can also assess production or staging traces without rerunning the agents that created them.
Plans and prices
Strands Evals SDK
Open-source Python SDK and CLI · install with pip · model-provider and trace-provider access may require separate credentials or services
Read 4 Oct 2026 · strandsagents.com
On the record
- Built-in evaluators
- Built-in evaluator groups cover quality, safety, multimodal responses, agentic behavior, and skill selection and instruction following.strandsagents.com · read 4 Oct 2026
- Custom evaluators
- Developers can implement domain-specific evaluation logic by extending the base Evaluator class.strandsagents.com · read 4 Oct 2026
- Default judge
- The quickstart states that evaluators use Amazon Bedrock with Claude as the default judge model, and that running its example requires credentials authorized to invoke Claude.strandsagents.com · read 4 Oct 2026
- Deterministic checks
- Deterministic evaluators run code-based checks without an LLM judge and include Equals, Contains, StartsWith, ToolCalled, StateEquals, and SkillInvoked.strandsagents.com · read 4 Oct 2026
- Evaluation levels
- Evaluators can assess a single output, tool call, trace, or full session, and multiple evaluators can be combined in one experiment.strandsagents.com · read 4 Oct 2026
- Experimental feature
- The red-teaming API is marked experimental and may change in a minor release.strandsagents.com · read 4 Oct 2026
- Intended users
- The SDK is presented for developers who want to validate agent behavior, measure improvements, and evaluate agents during development cycles.strandsagents.com · read 4 Oct 2026
- License
- The maker’s announcement identifies Strands Agents as an open-source project licensed under Apache License 2.0.strandsagents.com · read 4 Oct 2026
- Local and remote use
- The SDK can run evaluations against production or staging traces without rerunning the agents that produced them.strandsagents.com · read 4 Oct 2026
- Purpose
- Strands Evals measures agent behavior before shipping and after deployment by scoring outputs and trajectories, diagnosing failures, probing unsafe behavior, and simulating users and tools.strandsagents.com · read 4 Oct 2026
- SDK and CLI
- The package installs with pip as strands-agents-evals and provides both a Python API and the strands-evals command-line interface.strandsagents.com · read 4 Oct 2026
- Test generation and CI
- The CLI can generate experiments, validate experiment JSON, run evaluations, render reports, and diagnose sessions; its documentation shows validation and evaluation steps in a CI workflow.strandsagents.com · read 4 Oct 2026
- Trace integrations
- Trace providers fetch data from AWS CloudWatch Logs, Langfuse, and OpenSearch, with optional package extras for Langfuse and OpenSearch.strandsagents.com · read 4 Oct 2026
- Trace mapping
- Session mappers include support for Strands in-memory spans, LangChain OpenTelemetry spans, and OpenInference instrumentation such as Arize Phoenix.strandsagents.com · read 4 Oct 2026
