- Home
- LLM Evaluation Tools
- DecodingTrust
On the recordRead 11 Oct 202621 facts
DecodingTrust
On the record
- Assessment areas
- The benchmark covers toxicity, stereotype and bias, adversarial robustness, out-of-distribution robustness, privacy, adversarial demonstrations, machine ethics, and fairness.decodingtrust.github.io · read 11 Oct 2026
- Compute requirements
- The installation instructions note that the pip installation includes PyTorch and caution users to check CUDA compatibility with the PyTorch pip wheel.github.com · read 11 Oct 2026
- Container option
- The repository provides Docker instructions and a Docker image for running DecodingTrust.github.com · read 11 Oct 2026
- Content warning
- The project warns that its data contains model outputs that may be considered offensive.decodingtrust.github.io · read 11 Oct 2026
- Dataset
- The dataset is distributed under the CC BY-SA 4.0 license.decodingtrust.github.io · read 11 Oct 2026
- Deployment options
- The project documents Docker and Singularity or Apptainer container usage, including for HPC environments.github.com · read 4 Oct 2026
- Evaluation areas
- The benchmark covers toxicity, stereotype and bias, adversarial robustness, out-of-distribution robustness, privacy, adversarial demonstrations, machine ethics, and fairness.decodingtrust.github.io · read 4 Oct 2026
- Evaluation scripts
- The project provides evaluation scripts in subdirectories for its trustworthiness facets.decodingtrust.github.io · read 11 Oct 2026
- Installation
- The repository recommends cloning the project and installing it in editable mode with pip; it also describes standalone package installation.github.com · read 11 Oct 2026
- Intended users
- The project describes its resources as helping researchers and practitioners assess language model capabilities, limitations, and risks.decodingtrust.github.io · read 11 Oct 2026
- License
- The dataset and project are distributed under the CC BY-SA 4.0 license.decodingtrust.github.io · read 4 Oct 2026
- Model coverage limit
- The repository says its benchmark mainly focuses on GPT-3.5-turbo-0301 and GPT-4-0314 for consistent conclusions and results.github.com · read 4 Oct 2026
- Models
- The project says its evaluations mainly focus on GPT-3.5 and GPT-4, and it also supports causal LLMs hosted on Hugging Face or locally.github.com · read 4 Oct 2026
- Models evaluated
- The project says its evaluation mainly focuses on GPT-3.5 and GPT-4.decodingtrust.github.io · read 11 Oct 2026
- Ongoing contributions
- The team invites users to submit model generations and corresponding scores through GitHub pull requests and says the dataset is expected to evolve.decodingtrust.github.io · read 11 Oct 2026
- Platform support
- The repository says it supports the ppc64le architecture on IBM Power-9 platforms.github.com · read 11 Oct 2026
- Purpose
- DecodingTrust is a research project for assessing trustworthiness in GPT models and helping researchers and practitioners understand their capabilities, limitations, and deployment risks.decodingtrust.github.io · read 11 Oct 2026
- Reproducibility
- The benchmark uses timestamped GPT-3.5 and GPT-4 model versions to support consistent results and reproducibility.github.com · read 4 Oct 2026
