I Explored Respan Gateway: A Plugin-First, Open-Source LLM Gateway

Every time I connect an application to a new LLM provider, I run into the same problem: how much provider-specific code do I have to add? One SDK expects one request format, another provider uses a different API, and switching models can mean changing far more than a model name. Then come retries, fallback logic, usage tracking, and debugging when a request fails.

Most gateways answer this with the same pitch: one endpoint, many providers. Respan Gateway is more interesting for how it’s built than for that pitch. It is an open-source Rust gateway with a very small core, where nearly everything, including the providers, is a plugin. I went through the documentation and the launch post to see what that design means for a developer and what I’d check before relying on it.

One caveat first: the project describes itself as pre-alpha, and its config and APIs may still change.

Why the Architecture Is the Interesting Part

Respan says it ran its hosted gateway on LiteLLM since 2024. As customers needed different API-key settings, routes, and limits, its wrapper code grew to more than 100,000 lines. Then, this January, a routine LiteLLM upgrade caused intermittent connection errors and forced a rollback. Respan says it never found the cause and now backports upstream fixes by hand. That is the company’s own account, and I haven’t tested LiteLLM myself. It does explain the design goal: add features without breaking existing ones.

The result is a small core inspired by Nginx, responsible only for dispatching events. Providers, protocols (chat, real-time voice, MCP), modalities (images, video), and gateway features like routing and fallbacks are all plugins. A provider is a short file. Respan cites Mistral’s at 22 lines, and provider quirks, such as Groq reporting token usage in its own field, are separate plugins with their own tests.

Here is how a request moves through it:

A client adapter hands each HTTP or WebSocket connection to the first client format that recognizes it.
That format exposes the request as a two-way stream of typed events.
Plugins run in registration order. They can choose a provider, filter either stream, make extra calls, return a response, or refuse the request.
A provider format opens the upstream connection.

When client and provider speak the same API, and no plugin changes the payload, the original bytes and provider-specific fields pass through untouched. When the formats differ, requests translate through shared types for messages, tool calls, results, streams, and usage.

The trade-off: plugins are natively compiled Rust crates, not dynamically loaded or interpreted. Config-only template plugins cover common needs like adding a vendor or format without a rebuild. Custom logic means compiling it in. In the Hacker News launch thread, a commenter pointed out that this means a CI step instead of pulling a container image, and raised supply-chain concerns. Respan replied that compiled-in crates can at least be reviewed, and said it pulled its Docker image before launch because some APIs may still change. Official builds are prebuilt binaries on GitHub Releases.

Installing and Making a First Request

Respan Gateway can be installed in three ways: by downloading a binary from GitHub Releases, using Cargo, or installing the Python package from PyPI. Linux x86_64 is shown here (macOS, arm64, and Windows builds are also available), using the version shown in the release:

VERSION=v0.1.0
curl -fsSL "https://github.com/respanai/respan-gateway/releases/download/$VERSION/respan-gateway-$VERSION-x86_64-unknown-linux-gnu.tar.gz" | tar xz
sudo mv "respan-gateway-$VERSION-x86_64-unknown-linux-gnu/respan-gateway" /usr/local/bin/

Alternatively, install it with Cargo:

cargo install --git https://github.com/respanai/respan-gateway respan-gateway

Or install the Python package from PyPI:

pip install respan-gateway

A source build has no web UI unless you build it first. Check the current release before copying the version number.

With no config file, every provider key in your environment becomes a provider:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
respan-gateway

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
respan-gateway

The gateway prints the providers it found, the endpoints it serves, and a request to try. The model name in each request decides where it goes, and a prefix like openai/gpt-5.5 forces a provider. The docs’ curl example sends a request in OpenAI format to Gemini:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemini-3.6-flash", "messages": [{"role": "user", "content": "Hello!"}]}'

The same idea works from an SDK. Here, the OpenAI client talks to Claude:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
response = client.chat.completions.create(
    model="claude-sonnet-4-5-20250929",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

With no client keys configured, the gateway answers only on your own machine.

The documentation describes the startup output and request log available at /ui/requests. I have not included a screenshot or claimed a local test result here.

One File: gateway.toml

Everything beyond relaying a request is configured in one gateway.toml. respan-gateway init writes a commented starter from the keys in your environment. This block is adapted from the configure docs:

client_keys = ["env:GATEWAY_KEY"]

[providers.openai]
vendor = "openai"
key = "env:OPENAI_API_KEY"

[providers.anthropic]
vendor = "anthropic"
key = "env:ANTHROPIC_API_KEY"

[providers.gemini]
vendor = "gemini"
key = "env:GEMINI_API_KEY"

[routes.cheap-and-good]
to = ["openai/gpt-5.5", "anthropic/claude-sonnet-4-5-20250929", "gemini/gemini-3.6-flash"]
pick = "cheapest"
needs = ["tools"]
tries = 2

[routes.fast]
to = ["anthropic/claude-sonnet-4-5-20250929", "openai/gpt-5.5"]
hedge = "800ms"

A client asks for cheap-and-good as if it were a model name. These are the documented behaviors that matter:

pick sets the order candidates are tried in: first, cheapest (input price plus three times output price), fastest, weighted, or random.
needs filters candidates to models that list capabilities like tools, reasoning, vision, or pdf.
Fallback is on by default, so a failed candidate falls through to the next.
tries allows 1 to 10 attempts per candidate. Backoff starts at 250ms, doubles each time, and caps at 10s.
hedge races the candidates instead. Each starts after the delay, and the first answer wins.

Fallback models may differ in context length, tool support, or quality. Retries can waste tokens and add latency. Hedging deliberately sends more than one upstream request, which can increase cost.

Run respan-gateway check before starting (--online also calls each provider and MCP server). You can reload without a restart by sending SIGHUP, starting with --watch, or using the web UI. An invalid file is refused while the old config keeps serving. To see where a model would go and why, open /ui/api/routes/explain?model=cheap-and-good.

Before deployment, run respan-gateway check and inspect /ui/api/routes/explain?model=cheap-and-good to confirm the configuration and routing behavior in your environment.

For a failure test, configure an invalid key or unreachable provider as the first candidate, send a request, and inspect the request log to verify the actual retry and fallback behavior.

What It Can Proxy

The documented scope is chat, images, speech (including transcription), video, and real-time voice. The docs give examples like the Anthropic SDK calling a GPT model, and an OpenAI Realtime client talking to Gemini Live. It handles tools, images, streaming, and reasoning parameters across formats. Embeddings aren’t on that list, so retrieval pipelines will still need a separate path.

Sixteen providers are built in, including OpenAI, Anthropic, Gemini, Groq, DeepSeek, Mistral, xAI, OpenRouter, and Ollama. Any other OpenAI-, Anthropic-, or Gemini-style endpoint works with a base_url. Translation has limits, and Respan itself says not every feature carries over between providers. Test the exact models and parameters you use.

MCP Tools Behind the Gateway

Each [mcp.NAME] block puts an MCP server behind the gateway, with its credentials staying on the gateway. The tool_loop plugin lets any chat model use those tools mid-request. The gateway runs the tool, feeds the result back, and returns only the final answer, so the client needs no MCP support.

This is powerful and worth treating carefully. By default, any client key can make the model call any tool you offer, using the server’s credentials. Use deny lists for destructive tools, and per-key tools globs to limit what each client can reach.

Request Logs and Tracing

The web UI shows each request’s provider, latency, tokens, and cost. By default, it keeps the last 1,000 requests in memory. The request_log plugin can also append each finished request to a JSONL file, without bodies or keys.

The optional respan plugin sends each request to Respan as a trace. Its content setting defaults to true, meaning messages and replies are sent if your prompts contain sensitive data; set content = false to send metadata only.

Where It Fits

LiteLLM is the established self-hosted option, and Respan’s own history is built on it. A newer pre-alpha project can’t match its track record, and Respan’s pitch is mainly about extensibility.
OpenRouter is a hosted router with one API and one bill. It isn’t a competitor in the self-hosted sense, since it’s one of the gateway’s built-in vendors and you can route to it from here.
Respan’s hosted platform runs the gateway for you, adds observability and evals, and bills centrally. That is how the company makes money. The open-source binary is the self-run version, and you can combine them by adding a Respan key as a provider.

I’ve kept these comparisons to differences I could support. Check each project’s current docs before relying on them.

Before Using It in Production

Because it’s pre-alpha, I’d treat this as a proof of concept rather than a critical dependency for now. My checklist:

Stability: Expect config and API changes. Pin versions and read release notes.
Provider compatibility: Test streaming, tool calls, structured output, images, and any provider-specific parameters your feature uses.
Failure behavior: Simulate rate limits, timeouts, and bad keys. Confirm what retries, what falls back, and what happens when every candidate fails.
Latency: Measure end-to-end with and without the gateway, including time to first token.
Deployment: There’s no official Docker image right now, and custom plugins mean a rebuild. Plan for how you’ll build and ship it.
Security: Set client_keys before exposing it beyond localhost (it refuses to start on a public address without them). Review the auth code you depend on. A commenter on the launch thread questioned the project’s MCP OAuth implementation, and Respan said it used Claude to generate code but reviewed every line.
State: Conversation state for translated Responses API clients lives in one gateway’s memory, so multi-instance setups need clients pinned to the right instance.
Privacy: Decide what the request log and the respan plugin may record.

Verdict

Respan Gateway is worth a look if you want a small, extensible gateway with swappable providers and features as plugins, and you’re comfortable with compiled Rust extensions. It’s less suited to teams that need a drop-in container and a stable API today. Run it against your own models, break it on purpose, and then decide.

To explore it yourself, start with the GitHub repository, then the open-source gateway introduction and quickstart. For how it relates to the hosted offering, see the Respan AI Gateway overview.

Frequently Asked Questions

What is Respan Gateway?

An open-source Rust LLM gateway with a small core in which providers, protocols, and features are plugins. It translates between API formats and supports routing, retries, fallbacks, MCP tools, and a request log.

Is it open source?

Yes, under your choice of MIT or Apache-2.0. Confirm against the repository’s license files.

How do I install it?

Respan Gateway can be installed by downloading a binary from GitHub Releases, using cargo install --git https://github.com/respanai/respan-gateway respan-gateway, or installing the Python package from PyPI with pip install respan-gateway.