TensorZero

TensorZero is a free, open-source stack for LLM applications that combines a gateway with observability, optimization, evaluation, and experimentation. The gateway provides a unified API for model providers and supports routing, retries, fallbacks, prompt templates, schemas, and multi-step workflows. Traffic can be routed for A/B tests, with feedback attached to individual inferences or episodes. Documented integrations include Anthropic, AWS Bedrock, Azure, Google, OpenAI, OpenRouter, Together AI, vLLM, xAI, and OpenAI-compatible endpoints. TensorZero also supports the OpenAI SDK, OpenTelemetry, and clients for major programming languages. Inference and feedback data are stored in the database configured by the user and can be queried directly with SQL. Deployment options include Docker, Docker Compose, Kubernetes with Helm, or building from source. API key authentication can protect gateway endpoints other than status and health checks. The project is available on GitHub but is no longer maintained; its repository is archived and read-only. The FAQ describes the license as Apache 2.0.

TensorZero combines gateway, evaluation, observability, and experimentation capabilities in a self-hosted project. It remains available on GitHub, but it is no longer maintained and the repository is archived.

TensorZero may suit developers who want a self-hosted LLM gateway and can manage its deployment and data storage. Its maintenance status is important for teams evaluating ongoing use.

TensorZero is a self-hosted LLM gateway and operations stack for teams building applications across multiple model providers. It suits developers who want routing, evaluation, and inference data in infrastructure they control. Its free, broad toolset is appealing, but the archived project is a poor fit for systems that need ongoing maintenance.

Overview

TensorZero brings a unified API, gateway, and UI together with observability, optimization, evaluation, and experimentation. It can route requests among major providers and OpenAI-compatible APIs, while storing inference and feedback data in a database chosen by the user. That gives SQL-capable teams direct access to their operational data, but puts database administration and deployment in their hands.

The gateway includes retries, fallbacks, prompt templates, schemas, and budget controls. TensorZero reports sub-millisecond P99 latency overhead under extreme load, a useful target for latency-sensitive systems. The larger operational concern is its status: the project remains on GitHub but is no longer maintained, and its repository is archived and read-only.

Key features

Routing and experiments

Fallbacks and traffic routing for A/B tests help teams direct requests across providers and compare alternatives. Multi-step workflows support sequences beyond a single model call, and feedback can be tied to individual inferences or episodes. These capabilities make TensorZero more than a pass-through gateway, particularly for teams that want to connect application outcomes with model behavior.

Data, integrations, and security

TensorZero stores inference and feedback data in the configured database, which teams can query directly with SQL. Documented integrations include Anthropic, AWS Bedrock, Azure, Google AI Studio, OpenAI, OpenRouter, vLLM, and xAI, along with OpenAI-compatible APIs. The OpenAI SDK, OpenTelemetry, and clients for major programming languages broaden how teams can connect it to existing applications.

API key authentication can protect gateway endpoints other than status and health checks. Pseudonymous aggregate usage analytics exclude application data, including inference inputs and outputs, and can be disabled in settings. The Apache 2.0 license permits use as open-source software; questions can go to Slack or Discord, with team Slack or Teams support offered by email.

Pricing

TensorZero LLMOps platform: free (100% self-hosted · open-source). The plan includes the gateway and UI, fallback routing, budget controls, and virtual API keys. There is no paid hosted tier in this plan, so teams must run and maintain the software themselves. That price is a strong fit for developers able to operate their own stack, but free software is a less compelling bargain when its project is archived and receives no maintenance.

Platforms

TensorZero supports API and web use, Linux, and self-hosted deployment. The gateway and UI can be deployed with Docker, Docker Compose, Kubernetes and Helm, or built from source. This flexibility suits infrastructure teams that can manage a deployment; it offers little convenience to readers looking for a hosted service they do not have to operate.

Who it's for

TensorZero is best suited to technically capable teams that want a unified gateway across model providers, direct SQL access to inference data, and built-in experiments within their own infrastructure. Its pricing and controls also suit teams that need to manage budgets and virtual API keys. Organizations that require a maintained project, managed operations, or dependable ongoing support should look elsewhere.

Pros and cons

• Pro: A unified API, provider integrations, fallbacks, retries, and A/B routing bring several gateway tasks into one self-hosted stack.
• Pro: Inference and feedback data stay in the user's database and remain queryable with SQL.
• Pro: The free, Apache 2.0 software includes budget controls and virtual API keys.
• Con: The project is no longer maintained, and its GitHub repository is archived and read-only, creating a substantial upkeep concern.
• Con: Teams must deploy and operate the gateway, UI, and database themselves.

Alternatives

For a broader comparison, see LLM Gateway Software.

• Kong Gateway is a paid gateway with a free trial, including a no-credit-card 30-day trial with no gateway limits and 30 days of analytics retention; choose it if those trial terms better suit your evaluation.
• Bifrost offers a free-forever, self-hosted Apache 2.0 OSS plan and an Enterprise plan; consider it if you want to compare another self-hosted option.
• Eden AI charges for API calls at provider prices plus a 5.5% platform fee when purchasing credits, with no subscription; consider it if that usage-based gateway model fits your budget.
• Envoy AI Gateway is a free, self-managed open-source project with a local CLI for Linux and macOS; consider it if that deployment model fits your environment.
• Helicone is a freemium, open-source alternative with a Hobby plan; compare it if you want a different open-source option.
• LangSmith has a free Developer plan at no cost, with one seat and up to 5k base traces per month before pay-as-you-go charges; consider it if that trace allowance and seat-based plan suit your needs.
• LiteLLM offers a free-forever, self-hosted open-source plan with 100+ providers, virtual keys, spend tracking, budgets, and rate limits; consider it if those gateway controls and provider breadth are a priority.
• LLM Gateway has a free plan with 200+ models across 40+ providers and a 20-requests-per-minute limit on free models; consider it if that catalog and free-tier cap suit your traffic.

Verdict

Choose TensorZero if you can self-host it and want provider routing, experimentation, and SQL-queryable inference data without a software charge. Its integrated gateway and operations features are the main reason to consider it. Look elsewhere if ongoing maintenance is a requirement: the archived, unmaintained project makes that a serious tradeoff.