- Home
- LLM Evaluation Tools
- RAGChecker
On the recordRead 4 Oct 202614 facts
RAGChecker
Plans and prices
RAGChecker
Free
Apache-2.0 licensed open-source framework · installable with pip
Read 4 Oct 2026 · github.com
On the record
- Benchmark
- The repository says its benchmark was released and links a guide for using it.github.com · read 4 Oct 2026
- Fine-grained evaluation
- Its fine-grained evaluation uses claim-level entailment operations.github.com · read 4 Oct 2026
- Generator metrics
- Documented generator metrics include context utilization, noise sensitivity, hallucination, self-knowledge, and faithfulness.github.com · read 4 Oct 2026
- Input requirement
- For each query, the only required annotation is the ground-truth answer; generated responses and retrieved context are also accepted in the input format.github.com · read 4 Oct 2026
- Installation
- The maker documents installation with pip install ragchecker and requires downloading the spaCy en_core_web_sm model in its quick start.github.com · read 4 Oct 2026
- Integration
- The README documents an integration with LlamaIndex for evaluating RAG applications built with it.github.com · read 4 Oct 2026
- License
- The project is licensed under Apache-2.0.github.com · read 4 Oct 2026
- Model setup
- The example pipeline uses configurable extractor and checker models and documents an AWS Bedrock Llama 3 70B example.github.com · read 4 Oct 2026
- Pipeline metrics
- It provides overall metrics for the RAG pipeline and diagnostic metrics for retrieval and generation components.github.com · read 4 Oct 2026
- Purpose
- RAGChecker is an automatic evaluation framework for assessing and diagnosing retrieval-augmented generation systems.github.com · read 4 Oct 2026
- Retriever metrics
- Documented retriever metrics include claim recall and context precision.github.com · read 4 Oct 2026
- Security reporting
- The contribution guide asks users to report potential security issues to AWS/Amazon Security through its vulnerability reporting page rather than public GitHub issues.github.com · read 4 Oct 2026
- Support and contributions
- The project directs bug reports and feature requests to its GitHub issue tracker and welcomes pull requests.github.com · read 4 Oct 2026
- Usage
- The framework can be run through a CLI or Python API, with selectable metric groups or all metrics.github.com · read 4 Oct 2026
