Most Solana teams already pay for two RPC providers. Ask them why, and the answer is "redundancy." But look at how the second provider is actually wired in and you usually find the same pattern: one provider is hardcoded as the endpoint, and the other is a URL someone will paste in when things break. That isn't redundancy. It's a backup that requires a human to activate — and the human is usually asleep when it matters.
Real failover means traffic moves to a healthy provider automatically, in under a second, without anyone waking up. Getting there takes three things most setups are missing: a definition of "healthy" that goes beyond HTTP 200, a mechanism to route around unhealthy providers, and a way to bring them back safely. Here's how to think about each.
Manual failover is a backup, not redundancy
The problem with human-in-the-loop failover isn't that people are unreliable. It's the latency. A provider degrades at 2am. Alerts fire, someone acknowledges, logs in, confirms it's not their own bug, swaps the endpoint, redeploys, and waits for the change to propagate. That's minutes to tens of minutes of degraded or failed requests — and on a chain where a transaction has to land inside a narrow slot window, minutes is an eternity.
Automated failover collapses that loop to a health check and a routing decision. The tradeoff is that you have to encode "is this provider healthy?" as a rule a machine can evaluate continuously. That turns out to be the hard and interesting part.
"Is it up?" is the wrong question
The instinct is to failover when a provider returns errors. That catches the easy case — a provider that's hard-down, returning 500s or refusing connections. But the failures that actually cost you on Solana rarely look like errors:
-
Slot drift. A provider is answering every request with a clean
200 OK, but its view of the chain is several slots behind the network tip. YourgetAccountInforeturns real data — just stale data. Your uptime graph stays green the whole time. -
Commitment stalls. A node's
processedslot keeps advancing while itsconfirmedpipeline quietly freezes. Probe only the optimistic tip and it looks perfectly fresh, while everyconfirmedread serves increasingly old state. -
Rate limits. A
429isn't a fault — the provider is fine, you're just over your quota on that key for the moment. You want to shed load off it without declaring it down, because it'll have headroom again in seconds.
A failover system that only reacts to hard errors misses all three. The fix is to score providers on a continuous signal, not a binary one.
Health scoring: pick the best provider, not just an up one
Instead of a live/dead flag, give each provider a score in [0, 1], recomputed every probe cycle, from the signals that actually predict good responses:
- Latency — round-trip time on a lightweight probe.
- Error rate — fraction of recent probes that failed.
-
Slot freshness — how far behind the network tip the provider is, measured per commitment so a stalled
confirmedpipeline demotes the provider even whenprocessedlooks current. - Recent success rate — a fast-reacting signal so a provider that just started failing loses priority immediately.
The network tip itself is just the maximum slot across your providers — which has a blind spot: if they all stall together, the tip slides down with them and drift reads as zero. An optional external reference endpoint (probed, never routed to) closes that gap. The router then always sends reads to the highest-scoring provider and uses the sorted list as its failover order. You're not just avoiding dead providers; you're continuously steering toward the freshest, fastest one.
Circuit breakers: fail fast, recover safely
Scoring decides who's best. A circuit breaker decides who's excluded, and — just as important — when to trust a provider again. The pattern is three states:
- Closed — normal. All requests eligible.
- Open — the provider is excluded from routing. The circuit opens after N consecutive probe failures, or when the rolling error rate crosses a threshold.
- HalfOpen — after a cooldown, one probe is allowed through. Success closes the circuit and restores the provider; failure reopens it and resets the timer.
The cooldown is what stops flapping. Without it, a provider that fails one request, recovers, fails again, would thrash in and out of rotation and drag your latency with it. With it, a struggling provider gets a quiet interval to recover before it's trusted with real traffic again.
Two nuances matter here. Rate limits should not open the circuit. A 429 means fail this request over to a peer and demote the provider's score so routing sheds load — but keep it eligible, because opening its circuit for a full cooldown just dumps its traffic onto everyone else at the exact moment the system is under pressure. And client errors shouldn't touch provider health at all — a malformed request (a 400 or 404) is your bug, not the provider's, and one buggy client loop shouldn't be able to serially trip every provider's breaker.
One more safety valve: if every provider's circuit is open, route to all of them anyway. A degraded attempt beats refusing to try.
Choosing a routing strategy
Redundancy isn't one-size-fits-all. The right strategy depends on whether you're optimizing for reliability, cost, or raw latency:
- Best-score — every read goes to the healthiest provider right now. The sensible default for most production workloads.
- Failover-ordered — a strict primary → secondary hierarchy. The secondary only sees traffic when the primary's circuit opens. Ideal for a cheap-or-dedicated primary with a metered backup you don't want billed unless you need it.
- Weighted-random — spread load proportionally across providers of similar quality. This is how you stay under per-provider rate limits and split cost across keys.
- Parallel-race — send to all healthy providers at once, return the first success. Your latency tracks the fastest provider, at the cost of N× the request volume.
Wiring it up
You can build all of this yourself — background health probes, a per-provider scorer, a circuit-breaker state machine, and retry logic that knows a 429 from a -32005 "node is behind." Or you can drop a proxy in front of your providers that already does it.
That's what RPC Plane is: a small self-hosted binary you point your app at instead of a single provider URL. It probes each configured provider, scores them, trips circuit breakers, and routes around failures automatically — one config file, no application code changes. The routing and health-scoring docs cover the full tuning surface.
Either way, the takeaway is the same: two providers you fail over by hand is a backup. Two providers that a health-scored router keeps both live is redundancy. The difference only shows up at 2am — which is exactly when you can't afford to be the failover mechanism yourself.
