- Home
- LLM Gateway Software
- vLLM Semantic Router
On the recordRead 11 Oct 202627 facts
vLLM Semantic Router
On the record
- Agent use
- Agent harnesses connect to the Router inference endpoint while retaining responsibility for the agent loop, tools, and task state.github.com · read 10 Oct 2026
- API compatibility
- The Quickstart demonstrates requests through an OpenAI-style /v1/chat/completions endpoint.vllm-sr.ai · read 10 Oct 2026
- Client protocols
- The documentation lists OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages as client-facing protocols.vllm-sr.ai · read 11 Oct 2026
- Data retention
- The security guide says replay, response cache, memory, response history, service logs, and provider logs can retain request-derived data, and recommends setting retention and deletion policies for enabled stores.vllm-sr.ai · read 11 Oct 2026
- Deployment choices
- The documentation describes cloud, data center, edge, and enterprise hybrid deployment environments.vllm-sr.ai · read 10 Oct 2026
- Deployment limits
- The Router chooses a model or model pool, while the inference platform handles replica placement, batching, autoscaling, cache locality, and accelerator scheduling.vllm-sr.ai · read 10 Oct 2026
- Deployment locations
- The site says users configure where inference runs, including cloud, private, or edge environments.vllm-sr.ai · read 11 Oct 2026
- Installation
- The quickstart lists Linux, macOS, or WSL2, Python 3.10 or newer, and Docker as requirements, with Podman available as a Linux fallback.vllm-sr.ai · read 11 Oct 2026
- Local and remote models
- It can keep eligible sensitive or offline traffic on local models and allow selected requests to escalate to approved remote models.vllm-sr.ai · read 10 Oct 2026
- Model backends
- The site names vLLM, SGLang, and OpenAI API among the model backends it can route to.vllm-sr.ai · read 11 Oct 2026
- Model routing
- It can route requests by task, capability, modality, context length, and tool requirements across configured model backends.vllm-sr.ai · read 10 Oct 2026
- Open source license
- The public GitHub repository lists an Apache-2.0 license.github.com · read 10 Oct 2026
- Playground
- The project provides an online Playground at app.vllm-sr.ai/playground.github.com · read 10 Oct 2026
- Privacy policy
- The site describes routing private domain knowledge to open models on on-premises GPUs while allowing policy-eligible requests to use external frontier models.vllm-sr.ai · read 11 Oct 2026
- Product
- vLLM Semantic Router describes itself as an open, programmable decision layer for models and compute.vllm-sr.ai · read 11 Oct 2026
- Project team
- The project team page lists five maintainers responsible for project-wide technical and operational stewardship.vllm-sr.ai · read 11 Oct 2026
- Protocol limitations
- The compatibility matrix says custom free-form tools are not supported for Anthropic Messages, and audio input is supported only for Chat Completions.vllm-sr.ai · read 11 Oct 2026
- Purpose
- vLLM Semantic Router is an open, programmable decision layer that selects or combines models for each call according to policy.github.com · read 10 Oct 2026
