Three weeks after our verification layer finally stopped paging me at night, sales closed a customer whose contract had one non-negotiable line in it: dedicated compute. Their requests could never share backend instances with other tenants. I remember reading that clause and feeling almost smug. We had spent months on the hard parts. LangGraph orchestration, eval pipelines, a verification layer I had built twice because the first one was wrong. The agents were ready for this customer.
The routing was not.
Pointing their hostname at a dedicated pool meant editing gateway configuration, opening a pull request, waiting for review, and riding the Thursday release train. The customer could be contractually live on Tuesday. I sat in a planning meeting listening to us negotiate a go-live date around our own deployment calendar, and it occurred to me that we had built a platform where spinning up a new agent workflow took an afternoon but onboarding a paying customer took a sprint.
That is backwards. This article is about fixing it: a YARP gateway where the routing table lives in a database, refreshes at runtime across multiple replicas, and turns tenant onboarding from a deployment ceremony into a database write. And specifically, why this matters more for agent platforms than for almost any other kind of SaaS.
Why the Gateway Is the Control Point of an Agent Platform
Most writing about agent systems, including a good chunk of my own, focuses on what happens inside the orchestration graph. Tool calls, memory, evals, verification. The front door gets a paragraph, if that.
But tenants on an agentic platform differ from ordinary SaaS tenants in ways that all resolve at the gateway:
- Model tier. Tenant A pays for the frontier model, tenant B runs on the cheaper tier. Which backend pool a request lands on decides the unit economics of that request.
- Token budgets. A misbehaving client on a normal SaaS wastes your CPU. On an agent platform it burns real money per request. Rate limiting is not hygiene here, it is cost control.
- Isolation. Compliance-sensitive customers demand dedicated compute, sometimes dedicated model endpoints. That is a routing decision.
- Request duration. Agent loops run long. A gateway tuned for 200 ms CRUD calls will strangle a 90-second multi-step agent run with its default timeouts.
Every one of these is per-tenant policy, and the gateway is the one place that sees every request before any of it happens.
So why YARP and not a managed gateway product? Because YARP is a library, not an appliance. It runs inside a normal ASP.NET Core application, which means tenant resolution, rate limiting, and request enrichment are middleware you write in C#, tested like any other code, deployed like any other service. When tenant policy is genuinely core logic of your platform, and on an agent platform it is, I want that logic in my codebase, not in an appliance’s plugin model.
The cost is that you own it. I will be honest about what that means later.
One more decision to state up front, because it shapes the examples: the gateway in this article is .NET, and the agent backends behind it are Python. That is not a compromise, it is what a lot of real platforms look like. My orchestration layer is LangGraph, my gateway team thinks in ASP.NET Core, and the gateway genuinely does not care. HTTP is the contract.
The Architecture
+---------------------+
| Admin API |
| (validate, promote,|
| audit, rollback) |
+----------+----------+
|
v
+------------------------+
| Config Store (PG) |
| versioned snapshots |
| active_version ptr |
+-----+------------+-----+
| |
poll/notify poll/notify
| |
+-----------v--+ +--v-----------+
| Gateway #1 | | Gateway #2 |
| YARP + live | | YARP + live |
| routing table| | routing table|
+--+-----+--+--+ +--+-----+--+--+
| | | | | |
+-----------+ | +-----+------+ | +----------+
v v v v v
+-------------+ +-------------+ +-------------+ +-------------+
| Shared pool | | Dedicated | | Canary pool | | ... |
| (Bedrock) | | pool | | (new agent | | |
| LangGraph | | (tenant B) | | backend) | | |
+-------------+ +-------------+ +-------------+ +-------------+
The structural idea: the config store holds immutable, versioned routing snapshots, and each gateway replica’s in-memory routing table is a projection of whichever version is currently active. Onboarding a tenant, moving a tenant to dedicated compute, canarying a new backend: all of these are writes to the store followed by a pointer flip. No deploy anywhere in that sentence.
Most YARP articles stop at “poll a table every 30 seconds.” That works on stage one. It falls apart on the three problems that actually bit me: config that cannot be rolled back, replicas that disagree with each other, and a projection function nobody ever tested. Those three problems are the article.
The Config Store: Versions, Not Rows
My first version of this had one mutable tenant_routes table that the gateway polled. It worked until someone fat-fingered a destination URL, the gateway picked it up within 30 seconds, and one tenant's traffic went to a black hole. Rolling back meant reconstructing what the rows used to look like. From memory. During an incident.
Never again. Config versions are immutable, and “current config” is a pointer:
CREATE TABLE config_versions (
version_id BIGSERIAL PRIMARY KEY,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
created_by TEXT NOT NULL,
comment TEXT NOT NULL
);
CREATE TABLE tenant_routes (
version_id BIGINT NOT NULL REFERENCES config_versions(version_id),
tenant_id TEXT NOT NULL,
host_name TEXT NOT NULL,
path_pattern TEXT NOT NULL,
pool_name TEXT NOT NULL,
model_tier TEXT NOT NULL,
rate_limit_rpm INT NOT NULL,
timeout_seconds INT NOT NULL,
PRIMARY KEY (version_id, tenant_id, host_name, path_pattern)
);
CREATE TABLE pool_destinations (
version_id BIGINT NOT NULL REFERENCES config_versions(version_id),
pool_name TEXT NOT NULL,
dest_name TEXT NOT NULL,
address TEXT NOT NULL,
PRIMARY KEY (version_id, pool_name, dest_name)
);
CREATE TABLE active_config (
singleton BOOLEAN PRIMARY KEY DEFAULT TRUE CHECK (singleton),
active_version BIGINT NOT NULL REFERENCES config_versions(version_id),
promoted_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
This buys three things that a mutable table never can:
- Validation happens at promotion time, not at read time. The admin API validates a candidate version in full before flipping the pointer. The gateway never sees a half-written config, because a version is either promoted or it does not exist as far as the gateway is concerned.
- Rollback is a pointer flip. UPDATE active_config SET active_version = 41. Ten seconds into an incident, that sentence is worth the entire pattern.
- The audit trail is the data model. Who changed routing, when, and to what, without bolting on a logging framework.
Yes, this stores redundant rows across versions. Routing config is tiny. I will happily pay kilobytes for a rollback story.
The Refresh Service, Done Properly
YARP’s InMemoryConfigProvider has an Update method that atomically swaps the whole routing table at runtime. In-flight requests finish on the old table, new requests see the new one, and there is no window with no routes. That atomic swap is the load-bearing primitive of everything here.
Registration:
builder.Services.AddReverseProxy()
.LoadFromMemory(
routes: Array.Empty<RouteConfig>(),
clusters: Array.Empty<ClusterConfig>());
builder.Services.AddHostedService<RoutingRefreshService>();
The mistake I see in every example of this pattern, and made myself, is welding the database read, the projection to YARP config, and the Update call into one method inside the hosted service. That method is then untestable without a database, so it never gets tested, and it is precisely the code that decides where every request on your platform goes.
Pull the projection out. It is a pure function:
public static class RoutingProjection
{
public static (IReadOnlyList<RouteConfig> Routes,
IReadOnlyList<ClusterConfig> Clusters)
Project(IReadOnlyList<TenantRouteRow> tenantRoutes,
IReadOnlyList<PoolDestinationRow> destinations)
{
var routes = tenantRoutes.Select(t => new RouteConfig
{
RouteId = $"route-{t.TenantId}-{t.PoolName}",
ClusterId = $"cluster-{t.PoolName}",
Match = new RouteMatch
{
Path = t.PathPattern,
Hosts = new[] { t.HostName }
},
Metadata = new Dictionary<string, string>
{
["tenantId"] = t.TenantId,
["modelTier"] = t.ModelTier
}
}).ToList();
var clusters = destinations
.GroupBy(d => d.PoolName)
.Select(g =>
{
var timeout = tenantRoutes
.Where(t => t.PoolName == g.Key)
.Select(t => t.TimeoutSeconds)
.DefaultIfEmpty(100)
.Max();
return new ClusterConfig
{
ClusterId = $"cluster-{g.Key}",
LoadBalancingPolicy = "RoundRobin",
HttpRequest = new ForwarderRequestConfig
{
// agent runs are long; default timeouts kill them
ActivityTimeout = TimeSpan.FromSeconds(timeout)
},
Destinations = g.ToDictionary(
d => d.DestName,
d => new DestinationConfig { Address = d.Address })
};
}).ToList();
Validate(routes, clusters);
return (routes, clusters);
}
private static void Validate(
IReadOnlyList<RouteConfig> routes,
IReadOnlyList<ClusterConfig> clusters)
{
var clusterIds = clusters.Select(c => c.ClusterId).ToHashSet();
var orphan = routes.FirstOrDefault(
r => !clusterIds.Contains(r.ClusterId!));
if (orphan is not null)
throw new InvalidRoutingConfigException(
$"Route {orphan.RouteId} references missing cluster {orphan.ClusterId}");
var badDest = clusters
.SelectMany(c => c.Destinations!.Values)
.FirstOrDefault(d =>
!Uri.TryCreate(d.Address, UriKind.Absolute, out var u)
|| (u.Scheme != "http" && u.Scheme != "https"));
if (badDest is not null)
throw new InvalidRoutingConfigException(
$"Invalid destination address: {badDest.Address}");
}
}
Because it is pure, testing it is trivial, and these tests have caught real mistakes for me:
[Fact]
public void Route_referencing_missing_pool_is_rejected()
{
var routes = new[]
{
new TenantRouteRow("tenant-a", "tenant-a.platform.example",
"/api/{**catch-all}", "pool-that-does-not-exist",
"standard", 60, 120)
};
Assert.Throws<InvalidRoutingConfigException>(
() => RoutingProjection.Project(routes, Array.Empty<PoolDestinationRow>()));
}
The hosted service around it becomes thin, and its one non-negotiable rule fits in a catch block:
public class RoutingRefreshService : BackgroundService
{
private readonly InMemoryConfigProvider _provider;
private readonly IRoutingConfigRepository _repo;
private readonly ILogger<RoutingRefreshService> _log;
private long _appliedVersion = -1;
public RoutingRefreshService(
InMemoryConfigProvider provider,
IRoutingConfigRepository repo,
ILogger<RoutingRefreshService> log)
=> (_provider, _repo, _log) = (provider, repo, log);
protected override async Task ExecuteAsync(CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
try
{
var active = await _repo.GetActiveVersionAsync(ct);
if (active != _appliedVersion)
{
var snapshot = await _repo.GetSnapshotAsync(active, ct);
var (routes, clusters) = RoutingProjection.Project(
snapshot.TenantRoutes, snapshot.Destinations);
_provider.Update(routes, clusters);
_appliedVersion = active;
_log.LogInformation(
"Applied routing version {Version}: {Routes} routes, {Clusters} clusters",
active, routes.Count, clusters.Count);
}
}
catch (Exception ex)
{
// a failed refresh must never degrade the live table:
// keep serving the last known good config
_log.LogError(ex,
"Routing refresh failed; retaining version {Version}",
_appliedVersion);
}
await Task.Delay(TimeSpan.FromSeconds(10), ct);
}
}
}
If the config store is down, the gateway keeps routing on the last version it applied. A dead database should mean “we cannot change routing right now,” never “we stopped routing.”
The Problem Nobody Writes About: You Have More Than One Replica
Every YARP-dynamic-config article I have read, including the good ones, silently assumes a single gateway instance. Nobody runs a single gateway instance.
Run three replicas, each polling on its own clock, and promoting a new version creates a window where replica 1 routes tenant X to the new pool while replica 3 still routes them to the old one. For stateless request routing that window is usually harmless: both pools are serving the same API, and within one poll interval everything converges. The versioned design gives you exactly the vocabulary you need to reason about it: propagation delay is bounded by the poll interval, and you can watch convergence by exporting applied_version as a metric per replica. When all replicas report the same number, the fleet agrees.
I made two deliberate decisions here, and I want to defend the boring one.
First: I accepted eventual consistency instead of building coordination. The alternatives, distributed locks or a consensus step before applying config, add failure modes to the exact component that must not have failure modes. A bounded few-seconds skew between replicas is a much smaller problem than a gateway that can deadlock on its own config update.
Second: for faster propagation, I added a push channel but kept the poll. Postgres LISTEN/NOTIFY on promotion wakes the replicas immediately, and the 10-second poll remains as the safety net for missed notifications and freshly started replicas. Push for latency, poll for correctness.
The one case where the skew genuinely matters is moving a stateful workload, and agent platforms have those: long-running agent sessions, streaming responses, websockets. Yanking a tenant’s old pool out of the config mid-session drops those connections. The versioned store handles this gracefully: promote an intermediate version where the old pool’s destinations remain present but the tenant’s route points at the new pool, let existing sessions drain, then promote a final version that removes the old destinations. Migration becomes two pointer flips with a coffee in between, and each step is independently reversible.
Per-Tenant Policy, Not Just Per-Tenant Routing
Routing is table stakes. The reason to own the gateway is what you can attach to a request once you know whose it is.
Tenant resolution runs as ordinary middleware before YARP, and enriches the request:
public class TenantResolutionMiddleware
{
private readonly RequestDelegate _next;
public TenantResolutionMiddleware(RequestDelegate next) => _next = next;
public async Task InvokeAsync(HttpContext ctx, ITenantCache tenants)
{
var tenant = await tenants.ResolveByHostAsync(ctx.Request.Host.Host);
if (tenant is null)
{
ctx.Response.StatusCode = StatusCodes.Status404NotFound;
return;
}
ctx.Items["Tenant"] = tenant;
ctx.Request.Headers["X-Tenant-Id"] = tenant.Id;
ctx.Request.Headers["X-Model-Tier"] = tenant.ModelTier;
await _next(ctx);
}
}
Those two headers quietly delete tenant-resolution code from every LangGraph service behind the gateway. The graph reads X-Model-Tier and picks its Bedrock model accordingly; it never needs to know how tenants map to hostnames. Resolve once at the edge, trust it inside the private network boundary. One caveat I treat as a hard rule: this only holds if the gateway is the sole ingress and backends reject traffic from anywhere else. A trusted header on a reachable-from-anywhere backend is a spoofing invitation.
ITenantCache is exactly what it sounds like. Resolution runs on every request, so it is an in-memory cache with a short TTL over the store, refreshed alongside the routing table. A per-request database lookup at the front door is a latency tax you charge every tenant for no reason.
Then rate limiting, which on an agent platform I consider an economic control, not a courtesy. .NET’s built-in rate limiter partitions cleanly by tenant:
builder.Services.AddRateLimiter(options =>
{
options.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(ctx =>
{
var tenant = ctx.Items["Tenant"] as Tenant;
return RateLimitPartition.GetTokenBucketLimiter(
tenant?.Id ?? "anonymous",
_ => new TokenBucketRateLimiterOptions
{
TokenLimit = tenant?.RateLimitRpm ?? 10,
TokensPerPeriod = tenant?.RateLimitRpm ?? 10,
ReplenishmentPeriod = TimeSpan.FromMinutes(1),
QueueLimit = 0
});
});
options.OnRejected = async (ctx, ct) =>
{
ctx.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
await ctx.HttpContext.Response.WriteAsync(
"Rate limit exceeded for tenant.", ct);
};
});
The limit itself comes from the tenant’s row in the config store, which means raising a customer’s quota after an upsell is, once again, a database write. I have come to think of requests-per-minute at the gateway as a crude but effective proxy for tokens-per-minute at the model layer. It is not exact, real token budgeting needs accounting inside the agent runtime, but the gateway limit is the circuit breaker that caps the blast radius of a runaway client before the fine-grained accounting even wakes up.
And the pipeline order matters more than any individual piece: resolution, then rate limiting, then proxying.
app.UseMiddleware<TenantResolutionMiddleware>();
app.UseRateLimiter();
app.MapReverseProxy();
Routing as the Migration Mechanism
The dedicated-compute customer from the opening is where all of this converged. Here is how that onboarding actually looks with the pattern in place, and two more scenarios that fell out of it for free.
The dedicated pool. Their contract requires isolated compute. We stand up the pool, then create a new config version: their hostname routes to pool-dedicated-b, everyone else stays on pool-shared. Promote. The audit row says who did it and why. Elapsed time between "infrastructure ready" and "customer live": minutes. The Thursday release train still ran that week. It just did not have a customer chained to it.
The model-tier split. Premium tenants route to the pool backed by the frontier model, standard tenants to the cost-efficient tier. When a tenant upgrades, their model_tier and pool change in the next config version. Pricing tiers become routing rows, which is exactly the level of ceremony a pricing change deserves.
The canary migration. We rebuilt an agent backend and wanted real traffic on it before trusting it. One friendly tenant’s route moved to pool-canary in its own config version. We watched error rates and latency for that cluster specifically, per-cluster metrics come free with YARP's telemetry, and when the numbers held, moved the next tenant. Every step was one small version, and every step had a ten-second rollback. This is the calmest migration I have ever run, and the calm came entirely from the fact that each move was data, not a deploy.
Running the Whole Thing Locally with Ollama
I do not trust an architecture article I cannot run on my laptop, so here is the full loop with zero cloud dependencies: Postgres as the config store, the YARP gateway, and two Python agent backends talking to Ollama instead of Bedrock. The point of the demo is to watch routing change live while nothing redeploys.
The agent backend is deliberately minimal FastAPI. It identifies which pool served the request, which is how you will see routing move:
# agent/main.py
import os
import httpx
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
POOL_NAME = os.environ["POOL_NAME"]
MODEL = os.environ.get("MODEL", "llama3.1:8b")
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://ollama:11434")
class InvokeRequest(BaseModel):
prompt: str
@app.post("/api/agent/invoke")
async def invoke(req: InvokeRequest):
async with httpx.AsyncClient(timeout=120) as client:
resp = await client.post(
f"{OLLAMA_URL}/api/chat",
json={
"model": MODEL,
"messages": [{"role": "user", "content": req.prompt}],
"stream": False,
},
)
resp.raise_for_status()
answer = resp.json()["message"]["content"]
return {"served_by": POOL_NAME, "model": MODEL, "answer": answer}
The compose file wires everything together:
# docker-compose.yml
services:
postgres:
image: postgres:16
environment:
POSTGRES_DB: routing
POSTGRES_USER: gateway
POSTGRES_PASSWORD: gateway
volumes:
- ./sql/init.sql:/docker-entrypoint-initdb.d/init.sql
ports:
- "5432:5432"
ollama:
image: ollama/ollama:latest
volumes:
- ollama-models:/root/.ollama
ports:
- "11434:11434"
agent-shared:
build: ./agent
environment:
POOL_NAME: pool-shared
MODEL: llama3.1:8b
OLLAMA_URL: http://ollama:11434
depends_on: [ollama]
agent-dedicated:
build: ./agent
environment:
POOL_NAME: pool-dedicated
MODEL: llama3.1:8b
OLLAMA_URL: http://ollama:11434
depends_on: [ollama]
gateway:
build: ./gateway
environment:
ConnectionStrings__Routing: >-
Host=postgres;Database=routing;Username=gateway;Password=gateway
ports:
- "8080:8080"
depends_on: [postgres, agent-shared, agent-dedicated]
volumes:
ollama-models:
Pull the model once:
docker compose up -d
docker compose exec ollama ollama pull llama3.1:8b
init.sql seeds version 1: both pools registered as destinations, and tenant-a.localhost routed to pool-shared. Hit it through the gateway:
curl -s http://localhost:8080/api/agent/invoke \
-H "Host: tenant-a.localhost" \
-H "Content-Type: application/json" \
-d '{"prompt": "One sentence: what is a reverse proxy?"}'
{"served_by": "pool-shared", "model": "llama3.1:8b", "answer": "..."}
Now the actual demo. Move tenant-a to the dedicated pool by writing a new config version and flipping the pointer. No container restarts, no rebuilds:
# onboard-to-dedicated.sh
docker compose exec postgres psql -U gateway -d routing <<'SQL'
BEGIN;
INSERT INTO config_versions (created_by, comment)
VALUES ('ali', 'move tenant-a to dedicated pool');
-- copy current snapshot into the new version
INSERT INTO pool_destinations
SELECT currval('config_versions_version_id_seq'), pool_name, dest_name, address
FROM pool_destinations
WHERE version_id = (SELECT active_version FROM active_config);
INSERT INTO tenant_routes
SELECT currval('config_versions_version_id_seq'), tenant_id, host_name,
path_pattern, 'pool-dedicated', model_tier, rate_limit_rpm, timeout_seconds
FROM tenant_routes
WHERE version_id = (SELECT active_version FROM active_config)
AND tenant_id = 'tenant-a';
UPDATE active_config
SET active_version = currval('config_versions_version_id_seq'),
promoted_at = now();
COMMIT;
SQL
Run the same curl again within the poll interval:
{"served_by": "pool-dedicated", "model": "llama3.1:8b", "answer": "..."}
Routing moved while everything kept running. And the rollback drill, the one worth rehearsing before you need it, is a single statement pointing active_version back at the previous number. The first time I watched a tenant migrate between pools with a psql heredoc while the gateway logs just said "Applied routing version 2," the pattern stopped being a design document and became obvious.
Production Reality Check
I want to be precise about what this pattern does not give you, because the honest ledger is what decides whether you should build it.
You own a gateway now. A managed product brings a hardened edge, DDoS absorption, certificate automation, and a support contract. YARP brings none of that out of the box; those are your problems or your cloud provider’s. The split of responsibilities looks like this, and in production I run the hybrid in the middle column:
+---------------------+------------------+---------------------+------------------+
| Concern | Managed gateway | Hybrid (edge + YARP)| YARP alone |
+---------------------+------------------+---------------------+------------------+
| DDoS / edge | included | managed edge | yours |
| TLS / certs | included | managed edge | yours |
| Tenant-aware logic | plugin model | YARP, in C# | YARP, in C# |
| Dynamic per-tenant | varies, often | full control | full control |
| routing | awkward | | |
| Rate limit source | product config | your config store | your config store|
| Failure you debug | support ticket | your code + ticket | your code |
+---------------------+------------------+---------------------+------------------+
A thin managed edge in front for TLS and DDoS, with YARP as the tenant-aware layer behind it, keeps the parts you genuinely need to own and rents the parts you do not.
The refresh machinery is real code with a real maintenance bill. Versioned store, admin API, validation, projection, tests, the drain procedure. It is not enormous, but it is not free, and it is on the critical path of everything. Budget for it honestly.
A gateway bug you wrote is a full-platform outage. This is the sharpest edge. When I got the projection wrong in an early iteration, every tenant felt it simultaneously. The pure-function-plus-tests discipline and the version rollback exist because I learned this the loud way.
The trusted-header model has a boundary condition. X-Tenant-Id enrichment only works if the gateway is provably the only path to the backends. Network policy has to enforce what the code assumes.
And the pattern has a floor. If you have five tenants and onboard one a quarter, a static config file and a deploy is genuinely fine, and simpler is better. This earns its keep when tenant-aware routing is a living, frequently changing part of your platform. For an agent platform with real customers, it becomes that faster than you expect.
The Question Behind the Pattern
Strip away YARP and the pattern is really one question: which events in your business require a deployment, and which require only a decision?
Every time a routine commercial event, a new customer, an upsell, a migration step, is coupled to a release cycle, someone made an architectural choice, usually without noticing. Agent platforms feel this coupling harder than most software because the commercial events come with infrastructure consequences attached: isolation clauses, model tiers, token economics. The platform’s front door is where all of them land.
We spent months making our agents trustworthy: evals, verification, human-in-the-loop gates. It took one contract clause to show me that none of that mattered to a customer who could not be routed to it. The orchestration graph is the product, but the gateway is the promise that the product can be delivered to the next customer without an engineering ceremony. Build the front door so that saying yes to a customer never has to wait for Thursday.
Other Articles
- The Verification Layer Every AI Agent Needs (and How I Built One Twice)
- Event-Driven Systems in .NET, Python, and Go: A Practitioner’s Comparison
- Workflow Design Is a Thinking Discipline
Tags: Dotnet, Software Architecture, AI Agents, Microservices, API Gateway, Software Engineering, LLM
