- Home
- AI Agent Evaluation Tools
- OpenAgent Eval
On the recordRead 8 Oct 202620 facts
OpenAgent Eval
On the record
- Data handling
- The project says it runs on the user's machine without dashboards or accounts and that data never leaves the laptop.openagenthq.github.io · read 8 Oct 2026
- Embedders
- Sentence Transformers is the listed built-in embedder, and custom embedders can be added through provider base classes.openagenthq.github.io · read 7 Oct 2026
- Extensibility
- A plugin architecture supports custom metrics, providers, and report generators.openagenthq.github.io · read 7 Oct 2026
- Framework support
- The framework describes itself as compatible with LangChain, LlamaIndex, and custom RAG pipelines.openagenthq.github.io · read 8 Oct 2026
- Integrations
- Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider.openagenthq.github.io · read 8 Oct 2026
- License
- The project is licensed under Apache License 2.0.github.com · read 7 Oct 2026
- LLM integrations
- Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider.openagenthq.github.io · read 7 Oct 2026
- Local operation
- The project says it runs on the user's machine without telemetry or required network calls.openagenthq.github.io · read 7 Oct 2026
- Metrics
- Built-in metrics cover retrieval, generation, latency, and token cost.openagenthq.github.io · read 8 Oct 2026
- Notable dependency limit
- Some retrievers and embedders require extra dependencies, with an `[all]` package extra offered for the full set.openagenthq.github.io · read 8 Oct 2026
- Notable limitation
- Some retrievers and embedders require extra dependencies, such as chromadb, sentence-transformers, faiss-cpu, or qdrant-client.openagenthq.github.io · read 7 Oct 2026
- Offline use
- Built-in mock providers can run a complete evaluation locally without network calls or credentials.github.com · read 8 Oct 2026
- Purpose
- OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents.openagenthq.github.io · read 8 Oct 2026
- Reports
- Reports can be output in terminal, Markdown, HTML, or JSON formats, with built-in failure analysis.openagenthq.github.io · read 8 Oct 2026
- Requirements
- The installation guide lists Python 3.11 or later as a requirement.openagenthq.github.io · read 8 Oct 2026
- Retriever integrations
- Documented retrievers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock.openagenthq.github.io · read 7 Oct 2026
- Retrievers
- Documented retriever providers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock.openagenthq.github.io · read 8 Oct 2026
- Support
- The project directs users to GitHub Issues for bugs and GitHub Discussions for questions and ideas.github.com · read 8 Oct 2026
