- Home
- LLM Evaluation Tools
- AgentBench
On the recordRead 9 Oct 202622 facts
AgentBench
On the record
- Configuration
- The configuration system uses YAML with extended import, default, and overwrite keywords.github.com · read 9 Oct 2026
- Current tasks
- The current function-calling version supports alfworld, dbbench, knowledgegraph, os_interaction, and webshop tasks.github.com · read 9 Oct 2026
- Current version
- The repository says its current version is AgentBench FC, which uses function-calling prompts and integrates with AgentRL.github.com · read 9 Oct 2026
- Deployment
- The project documents a one-command setup for AgentBench FC tasks using Docker Compose.github.com · read 9 Oct 2026
- Environments
- The original benchmark includes eight environments, including operating systems, databases, knowledge graphs, house-holding, web shopping, and web browsing.github.com · read 9 Oct 2026
- Extensibility
- Users can add a task environment by implementing the Task class and specifying it in the configuration.github.com · read 9 Oct 2026
- FC task coverage
- AgentBench FC supports fully containerized deployment for alfworld, dbbench, knowledgegraph, os_interaction, and webshop.github.com · read 9 Oct 2026
- Hardware limit
- The README warns that the webshop environment requires approximately 16 GB of RAM to start.github.com · read 9 Oct 2026
- Knowledge graph dependency
- The KnowledgeGraph task depends on an online service that the README says is unstable, and it documents local deployment as an option.github.com · read 9 Oct 2026
- Known issue
- The README warns that the current alfworld implementation leaks memory and disk space until its task worker is restarted.github.com · read 9 Oct 2026
- License
- The repository is licensed under Apache License 2.0.github.com · read 9 Oct 2026
- Maintainer
- The THUDM GitHub profile identifies the group as THUKEG and lists FIT Building, Tsinghua University as its location.github.com · read 9 Oct 2026
- Model interfaces
- The framework supports local models served with FastChat and says models offered only through an API can be connected by implementing an interface in the Agent Client.github.com · read 9 Oct 2026
- Model setup
- The original quick start configures an OpenAI API key and gives gpt-3.5-turbo-0613 as its example agent.github.com · read 9 Oct 2026
- Original environments
- The original benchmark covers eight environments: operating system, database, knowledge graph, digital card game, lateral thinking puzzles, household tasks, web shopping, and web browsing.github.com · read 9 Oct 2026
- Other agents
- The README says users can replace the default agent by changing configuration parameters.github.com · read 9 Oct 2026
- Purpose
- AgentBench is a benchmark for evaluating the reasoning and decision-making abilities of LLMs acting as agents.arxiv.org · read 9 Oct 2026
- Requirements
- The README recommends Python 3.9 for dependency installation and requires Docker for the original quick-start setup.github.com · read 9 Oct 2026
