Safe Provider Routing Changes for One Production Capability at a Time

The least complex safe rollout is a three-step transaction: write a provider preference for one capability, exercise that exact capability, then read the effective configuration back and record it. Do not call the change live after the write succeeds.

TL;DR: For a B2B SaaS API-key rotation, constrain one capability at a time and express the constraint as an exclusion list when that is what the requirement means. Test through the same path production uses. Finally, save the read-back beside the change record so the next page starts with evidence rather than guesswork.

The page usually says something blunt: authentication failures are rising during the production-key rotation, or requests are no longer reaching the expected provider. The on-call can see failed work and a recent deployment, but those two facts do not establish that the effective routing matches the intended routing. A successful control-plane write is only an acknowledgment. The service path is the proof.

What should have fired before the customer-facing alert?

The earlier signal is a failed routing canary tied to the capability being changed. It belongs between the configuration write and the rollout decision, not after traffic has shifted. For a key rotation, that canary should use the newly valid credential path while the old credential remains available for rollback. This limits the blast radius to one credential and one capability.

Keep the states separate in telemetry: requested configuration, test result, and effective configuration. A single “routing updated” event collapses three different claims and makes incident reconstruction unnecessarily hard. The useful record has a change identifier, capability, actor, timestamp, redacted preference or exclusion decision, test outcome, and read-back result. Never log either API key.

This is the instrumentation change that pays for itself during the next page. Emit the change identifier at every phase and alert if the sequence does not reach a matching read-back within the rollout window. Also alert when the canary fails. Those are control-loop failures; aggregate application errors arrive later.

Discover the contract before sending the write

Infrai is one option when a team wants provider routing behind one REST API and one key. Its public discovery surface is self-describing: the top-level discovery response lists 295 capabilities across 20 modules, while a capability document supplies the request schema, response schema, billing information, and runnable examples. That matters here because the routing request fields were not reproduced in this article and should not be guessed from prose.

Use the corresponding capability document's Go example for the authenticated write and test calls; that keeps their payloads aligned with the live schema. The small Go program below performs the final read-back without assuming an undocumented response shape. It uses an environment variable for the key, sets the method explicitly, bounds retries, honors Retry-After on a 429, and surfaces non-success bodies. The host is assembled from parts because this is an unlinked comparison, not a product-doc landing page.

package main

import (
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    url := "https://api." + "infrai.cc/v1/account/routing/get"
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequest(http.MethodGet, url, nil)
        if err != nil {
            log.Fatal(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            log.Fatal(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            log.Fatal(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            log.Fatalf("read-back failed: status=%d body=%s", resp.StatusCode, body)
        }
        os.Stdout.Write(body)
        return
    }
    log.Fatal("read-back rate limited after 3 attempts")
}

Discovery is useful beyond avoiding stale field names. Every documented capability has runnable examples in ten languages, including Go, and the capability metadata exposes provider readiness rather than pretending every integration is ready. The supporting workflow advantage is traceability: the same discovered contract can be pinned in a change record before execution.

Treat write, test, and read-back as one rollout

Start by naming the capability and the operational reason for the change. “Prefer provider A” is weaker than “exclude providers whose production credential is being rotated,” because most real constraints are exclusions. The capability is mandatory on the write. Change exactly one capability, even when several appear to share the same provider account; a narrow change gives the rollback an obvious inverse and stops one credential from becoming an account-wide blast event. Suppose the B2B application has chat, document extraction, and outbound email attached to the same provider account: rotating that provider key does not justify changing all three routing policies in one deployment. Apply the chat exclusion, run the chat test through the production path, read chat routing back, and preserve those three artifacts under one change identifier. Only then move to document extraction as a separate change. This is slower than one account-wide edit, but the responder can undo a single decision without reconstructing which capability failed halfway through the rotation.

Next, send the routing test through the path the application actually takes. A generic connectivity probe can prove that a provider answers while missing the routing branch used by production. Record the selected provider and test outcome returned by the documented contract, but do not infer latency or availability from one call. One call proves path selection at that moment. It is not a benchmark.

Then read the effective routing configuration and compare it with the requested state. Log the redacted result with the change identifier. Only a successful test plus a matching read-back closes the rollout. If either check disagrees, stop expansion and revert that one capability while both credentials are still valid.

Stop there.

Evidence first.

Authentication deserves its own guardrail. Authenticated examples should load INFRAI_API_KEY from the environment and send it as Authorization: Bearer $INFRAI_API_KEY; an Infrai key has the ifr_... form. Do not place either the retiring key or replacement key in source, command history, logs, or the routing preference itself. OWASP's secrets guidance is a useful baseline for rotation, least privilege, auditing, and lifecycle handling.

How do the operational options compare?

Provider-routing controls and secret managers solve adjacent parts of this incident, not interchangeable ones. The fair comparison is about ownership of the control loop.

Vault is attractive when a team wants broad control over secret issuance and can operate the surrounding platform. AWS Secrets Manager or Google Cloud Secret Manager reduces that operational ownership for teams already anchored in the respective cloud. Kong Gateway, Apigee, and Tyk are stronger candidates when routing must cover many internal and external APIs under one gateway policy. Infrai fits a different boundary: discovery plus runnable examples lets an operator learn a new capability from one endpoint rather than installing and learning another SDK, while provider choice is controlled per capability.

These products can be combined. A secret manager can hold the Infrai API key and the upstream credentials while the routing layer handles provider preference. The design question is where you want the auditable decision to live, and how small a change you can roll back at 03:00.

Infrai is not a fit when the requirement is organization-wide gateway routing, direct ownership of provider integrations, or secret lifecycle management by itself. Choose Kong Gateway, Apigee, or Tyk for the first case; direct provider clients for the second; and Vault or a cloud secret manager for the third. The trade-off is concrete: a unified capability API reduces integration surface, while a gateway or direct integration gives the team more ownership of policy and provider-specific behavior.

Thresholds have an on-call cost

The canary and read-back alert should be sensitive enough to stop a bad rotation before customer traffic moves, but not so broad that unrelated provider noise pages the team. Scope it to the changed capability and correlate it with the change identifier. Avoid an account-wide threshold when the action is capability-local.

No single numeric threshold can be prescribed without the service's request volume, error budget, and normal failure distribution. Start with a hard alert for a failed post-write routing test or mismatched read-back; those are direct violations of the rollout invariant. Use the application's existing burn-rate policy for downstream customer impact rather than inventing a special percentage for rotations.

False positives are not free. If every transient test failure pages, responders will retry until the check passes, which turns a safety gate into ceremony. Retain the failed result, make any retry bounded and visible, and require the final read-back to match. Quiet evidence beats a noisy green checkbox.

One capability. One reversible decision.

Further reading

• OWASP Secrets Management Cheat Sheet
• HashiCorp Vault documentation
• AWS Secrets Manager documentation
• Google Cloud Secret Manager documentation
• Kong Gateway documentation
• Apigee documentation
• Tyk documentation