- Home
- LLM Evaluation Tools
- Inspect AI
Inspect AI
What it is
Inspect AI is a free, open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. Evaluations assemble datasets, solvers, and scorers, with more than 200 pre-built evaluations available. Inspect View provides web monitoring and visualization, while a VS Code extension lets developers browse logs, run and debug tasks, and configure options. The framework supports more than 20 model providers and local inference through Hugging Face, vLLM, and SGLang. Its tools include web search, code execution, text editing, computer use, and web browsing, plus MCP tools. Agents from OpenAI Agents SDK, Pydantic AI, LangChain, Claude Code, Codex CLI, and Gemini CLI can be integrated. Untrusted model code can run through Docker, Kubernetes, Modal, Proxmox, Vagrant, and other extension-supported systems. Inspect is self-hosted; installation uses PyPI, and development and runs require access to a model.
For and against
For
- Open-source and free to use
- More than 200 pre-built evaluations
- Supports over 20 model providers
- VS Code logging and debugging tools
- Multiple sandbox deployment options
Against
- Self-hosted deployment required
- Evaluation runs require model access
- Installation uses Python’s PyPI workflow
The verdict
Inspect AI brings datasets, solvers, scorers, visualization, and editor tools into one evaluation framework. Its broad model, agent, tool, and sandbox integrations support varied testing setups, provided your team supplies hosting and model access.
Who it suits
Inspect AI suits developers and teams evaluating language models or agents across varied tasks. It fits organizations wanting self-hosted evaluations, provider integrations, custom metrics, and safety workflows.
Plans and prices
Inspect AI
Open-source framework for large language model evaluations
Read 4 Oct 2026 · inspect.aisi.org.uk
On the record
- Agent integrations
- Inspect can integrate agents built with OpenAI Agents SDK, Pydantic AI, and LangChain, as well as CLI agents such as Claude Code, Codex CLI, and Gemini CLI.inspect.aisi.org.uk · read 4 Oct 2026
- Editor integration
- The Inspect VS Code extension provides tools for browsing evaluation logs, running and debugging tasks, and configuring options.inspect.aisi.org.uk · read 4 Oct 2026
- Evaluation results
- Inspect View is a web-based tool for monitoring and visualizing evaluations.inspect.aisi.org.uk · read 4 Oct 2026
- Installation requirement
- The getting-started instructions install Inspect from PyPI with pip install inspect-ai and say evaluation development and runs also require access to a model.inspect.aisi.org.uk · read 4 Oct 2026
- Model integrations
- Inspect has built-in support for more than 20 model providers and local inference with Hugging Face, vLLM, and SGLang.inspect.aisi.org.uk · read 4 Oct 2026
- Provider examples
- Documented provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Hugging Face, vLLM, Ollama, and SGLang.inspect.aisi.org.uk · read 4 Oct 2026
- Purpose
- Inspect is an open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding.inspect.aisi.org.uk · read 4 Oct 2026
- Reusable components
- Evaluations combine datasets, solvers, and scorers, and can use more than 200 pre-built evaluations.inspect.aisi.org.uk · read 4 Oct 2026
- Sandboxing
- The framework supports running untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems through an extension API.inspect.aisi.org.uk · read 4 Oct 2026
- Tools and MCP
- Inspect includes standard tools for web search, code execution, text editing, computer use, and web browsing, and supports MCP tools.inspect.aisi.org.uk · read 4 Oct 2026
