- Home
- AI LLM Evaluation Tools
- Rhesis AI
Rhesis AI
Potsdam, Germany
What it is
Rhesis AI is an open-source collaboration layer for developing and evaluating AI agents. Domain experts, engineers, and product managers can share a workspace to review test cases and agent versions, add annotations, and record expectations. Teams can generate test sets from application requirements or constraints, including single-turn prompts and multi-turn conversations. Rhesis runs these tests against an application endpoint, evaluates responses with configured metrics, and records results in test runs. Rhesis Tracing uses OpenTelemetry to capture LLM calls, tool invocations, and retrieval operations in testing or production. Evaluation options include offline evaluation, online trace metrics, LLM-as-a-judge, custom metrics, safety evaluations, and adversarial red-teaming. Model connections through LiteLLM include cloud and local providers such as OpenAI, Anthropic, Google Gemini, Azure OpenAI, Mistral, Ollama, and vLLM. Teams can use DeepEval and Ragas metrics or import Garak probes. The REST API supports test-set management, test runs, results, and trace inspection for integrations and CI/CD. Rhesis can run as a managed service, locally with Docker, or self-hosted; the site says it is developed in Germany and hosted in the EU.
For and against
For
- Shared workspace connects experts, engineers, and product managers.
- Generates single-turn and multi-turn test sets.
- OpenTelemetry tracing covers calls, tools, and retrieval.
- Supports cloud and local model providers through LiteLLM.
- Can run managed, locally with Docker, or self-hosted.
Against
- No paid plan price is listed in the provided facts.
- Security issues must be reported through the support system.
- The team cannot currently offer security bounties or rewards.
The verdict
Rhesis combines collaborative test creation, evaluation, tracing, and integrations for AI agent development. Its self-hosted open-source edition and cloud free plan are listed at no charge, though the cloud allowance is unspecified.
Who it suits
Rhesis is intended for domain experts, engineers, and product managers working on AI agents. It may suit teams that need shared test workflows, model integrations, tracing, or security evaluations.
Plans and prices
Rhesis Cloud Free
Start free on Rhesis Cloud with a monthly allowance; exact allowance not stated on the opened pricing page.
Read 30 Sep 2026 · rhesis.ai
Self-hosted open source edition
Nothing metered.
Read 30 Sep 2026 · rhesis.ai
On the record
- API and CI/CD
- The REST API supports managing test sets, triggering test runs, fetching results, and inspecting traces for custom integrations and CI/CD pipelines.docs.rhesis.ai · read 30 Sep 2026
- Deployment and data location
- Rhesis can run as a managed service, locally with Docker, or self-hosted; the site says it is developed in Germany and hosted in the EU.docs.rhesis.ai · read 30 Sep 2026
- Evaluation integrations
- Rhesis supports DeepEval and Ragas metrics and can import Garak probes as test sets for adversarial scanning.docs.rhesis.ai · read 30 Sep 2026
- Evaluations
- Rhesis runs test sets against an application endpoint, evaluates responses using configured metrics, and records results in test runs.docs.rhesis.ai · read 30 Sep 2026
- Intended users
- Rhesis describes itself as built for experts, engineers, and product managers working on AI agents.rhesis.ai · read 30 Sep 2026
- Model integrations
- Rhesis supports cloud and local model providers through LiteLLM, including OpenAI, Anthropic, Google Gemini, Mistral, Cohere, Groq, Together AI, Ollama, and vLLM.docs.rhesis.ai · read 30 Sep 2026
- Product
- Rhesis is an open-source collaboration layer that brings domain-expert knowledge into AI agent development.rhesis.ai · read 30 Sep 2026
- Security testing
- Rhesis integrates Garak to probe applications for issues such as prompt injection, data leakage, and harmful content generation.docs.rhesis.ai · read 30 Sep 2026
- Support
- The getting-started page points users to in-app support, documentation, a support email, GitHub, and Discord.docs.rhesis.ai · read 30 Sep 2026
- Team workflow
- Domain experts, engineers, and product managers can share a workspace for reviews, annotations, and expectations attached to test cases and agent versions.rhesis.ai · read 30 Sep 2026
- Test generation
- Rhesis can generate test sets from application requirements or constraints, including single-turn prompts and multi-turn conversations.docs.rhesis.ai · read 30 Sep 2026
- Tool integrations
- The integrations page lists Notion, GitHub, Jira, GitLab, Asana, Azure DevOps, Linear, and Shortcut as tool connections.docs.rhesis.ai · read 30 Sep 2026
- Tracing
- Rhesis Tracing uses OpenTelemetry to capture LLM calls, tool invocations, and retrieval operations in test or production mode.docs.rhesis.ai · read 30 Sep 2026
