I Built a Small API Gateway With Real Production Problems — On Purpose

Most gateway tutorials stop at "here's how you route a request." That's the easy 20%. The hard part is what happens when a client hammers you with requests, a downstream service falls over mid-traffic, or you're staring at a 500 trying to figure out which of your four services actually caused it.

I wanted to build something that hits those problems on purpose, so I put together spring-gateway-sample: a public gateway, an api-server that fans out to two downstream services, and a full observability stack sitting behind all of it. It's not a real product and never will be. But I tried to make it behave like one — including the annoying bits, like config tradeoffs and races that most demos just quietly ignore.

Stack, for context: Spring Boot 4.1, Spring Cloud Gateway on WebFlux, Resilience4j, Redis, Postgres, Keycloak, Prometheus/Grafana/Tempo/Loki, and a small Vue 3 app for throwing traffic at it from a browser.

The system, in one request

Browser (Vue traffic simulator)
   │  Keycloak PKCE login + API key
   ▼
Gateway  ── JWT + API-key auth, Redis rate limiting ──▶  routes to
   │
   ▼
api-server ── WebClient delegation, circuit breakers, Caffeine cache ──▶
   │                                                    │
   ▼                                                    ▼
product-service                                  pricing-service
(JPA / Postgres)                                 (JPA / Postgres)

Every hop re-validates the JWT on its own — defense in depth, so the gateway isn't the single thing standing between the internet and the data. The gateway also checks an API key on top, because a JWT tells you who the user is, not which client application is calling on their behalf. You need that second identity if you want per-client rate limits or the ability to revoke one app's access without touching anyone else's.

Two checks, one specific order

Every request needs a Keycloak JWT and an API key, and the order they're checked in isn't an accident:

1. Missing or expired JWT → 401, before the API key is even looked at.
2. Valid JWT, bad API key → 401, but a different error code.
3. Both valid, wrong role → 403.

Why bother with the ordering? Because "you're not authenticated at all" and "you're a real client with a bad key" are two different incidents from a security team's point of view, and the response should let a client tell them apart. Errors come back as RFC 9457 Problem Details (application/problem+json) with a custom code field, so callers can branch on TOKEN_EXPIRED vs INVALID_API_KEY instead of regex-matching a human-readable message. Every service uses the same pattern — a @RestControllerAdvice that maps each exception to a status plus a machine code:

@RestControllerAdvice
public class ProblemDetailExceptionHandler {

    @ExceptionHandler(ProductNotFoundException.class)
    public ResponseEntity<ProblemDetail> handleNotFound(ProductNotFoundException ex) {
        return problemResponse(HttpStatus.NOT_FOUND, ex.getMessage(), "PRODUCT_NOT_FOUND");
    }

    @ExceptionHandler(DataIntegrityViolationException.class)
    public ResponseEntity<ProblemDetail> handleDuplicateSku(DataIntegrityViolationException ex) {
        return problemResponse(HttpStatus.CONFLICT, "A product with this SKU already exists", "DUPLICATE_SKU");
    }

    private ResponseEntity<ProblemDetail> problemResponse(HttpStatus status, String detail, String code) {
        ProblemDetail problemDetail = ProblemDetail.forStatusAndDetail(status, detail);
        problemDetail.setProperty("code", code);
        return ResponseEntity.status(status)
                .contentType(MediaType.APPLICATION_PROBLEM_JSON)
                .body(problemDetail);
    }
}

Which gets you a response like this — a client can switch on code instead of parsing prose:

{
  "type": "about:blank",
  "title": "Not Found",
  "status": 404,
  "detail": "Product 8f21 not found",
  "code": "PRODUCT_NOT_FOUND"
}

API keys are never stored raw, for what I hope are obvious reasons. The gateway hashes them with HMAC-SHA256 plus a server-side pepper and only persists the digest (api_client.key_hash, unique-indexed so lookup is still O(1)):

@Component
public class ApiKeyHasher {

    private static final String HMAC_ALGORITHM = "HmacSHA256";
    private static final String RAW_KEY_PREFIX = "ak_";

    private final SecretKeySpec pepperKey;

    public ApiKeyHasher(@Value("${app.security.api-key.pepper:dev-only-api-key-pepper-change-me}") String pepper) {
        this.pepperKey = new SecretKeySpec(pepper.getBytes(StandardCharsets.UTF_8), HMAC_ALGORITHM);
    }

    public String digest(String rawKey) {
        try {
            Mac mac = Mac.getInstance(HMAC_ALGORITHM);
            mac.init(pepperKey);
            return HexFormat.of().formatHex(mac.doFinal(rawKey.getBytes(StandardCharsets.UTF_8)));
        } catch (GeneralSecurityException e) {
            throw new IllegalStateException("Failed to compute API key digest", e);
        }
    }

    public String generateRawKey() {
        byte[] randomBytes = new byte[32];
        SECURE_RANDOM.nextBytes(randomBytes);
        return RAW_KEY_PREFIX + Base64.getUrlEncoder().withoutPadding().encodeToString(randomBytes);
    }
}

One thing I want to be upfront about: rotating a key doesn't retroactively kill an already-cached validation in Redis. The cache TTL (app.security.api-key.cache-ttl: 60s) is effectively the worst-case revocation delay. That's a tradeoff, not a bug I forgot to fix — a 60-second window is fine for a demo, might not be fine for you, and either way it's better to say it out loud than bury it.

Rate limiting: three buckets, not one

Most gateway tutorials show you a rate limiter, singular. This one checks three, in sequence, each its own Redis token bucket:

The route bucket protects a downstream from aggregate load, full stop, regardless of who's calling. The per-client bucket stops one API key from hogging everything for itself. The per-IP bucket catches the case where abuse is spread across a bunch of keys but all coming from one place. And if a later bucket in the chain rejects, whatever got consumed from the earlier buckets gets refunded — otherwise a request that only fails the IP check would still have burned a client-bucket token for nothing. Here's the actual filter doing all three checks with refund-on-reject:

private Mono<Boolean> consumeSequentially(
        String routeId, String routeKey, RateLimitProperties.Bucket routeBucket,
        String clientKey, RateLimitProperties.Bucket clientBucket,
        String ipKey, RateLimitProperties.Bucket ipBucket) {
    return rateLimiter.tryConsume(routeKey, routeBucket).flatMap(routeAllowed -> {
        recordOutcome(routeId, RateLimitScope.ROUTE, routeAllowed);
        if (!routeAllowed) {
            return Mono.error(new RateLimitExceededException(RateLimitScope.ROUTE));
        }
        return rateLimiter.tryConsume(clientKey, clientBucket).flatMap(clientAllowed -> {
            recordOutcome(routeId, RateLimitScope.CLIENT, clientAllowed);
            if (!clientAllowed) {
                return rateLimiter.refund(routeKey, routeBucket)
                        .then(Mono.error(new RateLimitExceededException(RateLimitScope.CLIENT)));
            }
            return rateLimiter.tryConsume(ipKey, ipBucket).flatMap(ipAllowed -> {
                recordOutcome(routeId, RateLimitScope.IP, ipAllowed);
                if (!ipAllowed) {
                    return rateLimiter.refund(routeKey, routeBucket)
                            .then(rateLimiter.refund(clientKey, clientBucket))
                            .then(Mono.error(new RateLimitExceededException(RateLimitScope.IP)));
                }
                return Mono.just(true);
            });
        });
    });
}

Every check also bumps a Micrometer counter, allowed or rejected, which is what feeds the Grafana panel I'll get to later:

private void recordOutcome(String routeId, RateLimitScope scope, boolean allowed) {
    meterRegistry.counter(
            "gateway.rate_limit.requests",
            "route", routeId,
            "scope", scope.name().toLowerCase(),
            "outcome", allowed ? "allowed" : "rejected"
    ).increment();
}

I want to flag one thing before someone else does: this is three separate Redis round trips, not one atomic Lua script. There's a real (small) race window where two concurrent requests could both read "1 token left" and both get through. The Lua-script version closes that, but it's more code to understand for a demo whose entire point is to be understandable. I'd rather ship the simpler version and tell you about the race than pretend it isn't there.

When a downstream dies: circuit breakers with a real fallback

api-server wraps every GET to product-service and pricing-service in a Resilience4j circuit breaker backed by a Caffeine cache. Closed breaker, you get live data (and it gets cached on the way past). Open breaker — downstream's failing or slow — you get the last cached value instead of an error. The mechanics live in one base class every downstream client extends:

protected <T> Mono<GetResult<T>> getWithFallback(String cacheKey, Mono<T> call) {
    Mono<T> retried = call.retryWhen(Retry.max(1).filter(WebClientRequestException.class::isInstance));
    Mono<GetResult<T>> primary = retried
            .doOnNext(value -> cache.put(cacheKey, new CachedEntry(value, Instant.now())))
            .map(value -> (GetResult<T>) new GetResult.Live<>(value));
    ReactiveCircuitBreaker circuitBreaker = circuitBreakerFactory.create(circuitBreakerName);
    return circuitBreaker.run(primary, ex -> fallback(cacheKey, ex));
}

@SuppressWarnings("unchecked")
private <T> Mono<GetResult<T>> fallback(String cacheKey, Throwable ex) {
    if (ex instanceof DownstreamClientErrorException clientError) {
        return Mono.error(clientError); // a 4xx is a business rejection, not a circuit failure
    }
    CachedEntry cached = cache.getIfPresent(cacheKey);
    if (cached != null) {
        String reason = ex instanceof CallNotPermittedException
                ? serviceCode() + "_CIRCUIT_OPEN"
                : serviceCode() + "_CALL_FAILED";
        return Mono.just(new GetResult.Degraded<>((T) cached.value(), reason, cached.cachedAt()));
    }
    return Mono.error(new DownstreamUnavailableException(serviceName, ex));
}

Worth pointing at that first if: a downstream 4xx gets rethrown as-is and never counted as a circuit failure. "This product doesn't exist" and "the product service is down" are not the same problem, and if you let the first one trip the breaker, a normal stream of 404s starts serving stale data to everyone. That's the kind of bug that's obvious in hindsight and easy to write by accident.

A degraded response comes back looking like this:

{
  "data": { "...": "..." },
  "meta": { "reason": "PRODUCT_SERVICE_CIRCUIT_OPEN" }
}

meta.reason is what makes this a demo you can actually watch happen, instead of a diagram you have to take on faith — you (or a human staring at the Vue traffic simulator) can see the exact moment a service degrades, not just a latency graph doing something suspicious. No cached value yet, and it's an honest 503. Mutations don't get a fallback at all — a write either goes through or it fails loudly, because pretending it succeeded is worse than admitting it didn't:

private <T> Mono<T> mutationFallback(Throwable ex) {
    if (ex instanceof DownstreamClientErrorException clientError) {
        return Mono.error(clientError);
    }
    return Mono.error(new DownstreamUnavailableException(serviceName, ex));
}

To trigger this without actually killing a container mid-demo, product-service and pricing-service both accept a dev-only ?mode=fail / ?mode=slow query param. Hit an endpoint with it and watch the breaker trip live in Grafana.

One more thing that cost me real debugging time: Boot 4.1 dropped WebClient's autoconfiguration in favor of RestClient for blocking use cases, so there's no free WebClient.Builder bean sitting around anymore. Each downstream client gets built by hand — and each one has to explicitly forward the inbound bearer token, because product-service and pricing-service check their own JWTs too:

@Bean
public WebClient productServiceWebClient(DownstreamProperties properties) {
    return WebClient.builder()
            .baseUrl(properties.productService().baseUrl())
            .filter(forwardBearerToken())
            .build();
}

private ExchangeFilterFunction forwardBearerToken() {
    return ExchangeFilterFunction.ofRequestProcessor(request -> ReactiveSecurityContextHolder.getContext()
            .map(SecurityContext::getAuthentication)
            .filter(JwtAuthenticationToken.class::isInstance)
            .map(JwtAuthenticationToken.class::cast)
            .map(auth -> ClientRequest.from(request)
                    .headers(headers -> headers.setBearerAuth(auth.getToken().getTokenValue()))
                    .build())
            .defaultIfEmpty(request));
}

I found this the fun way — everything downstream started 401ing, and it took a minute to realize the gateway had already done its job validating the caller, api-server just wasn't passing the token along.

The infrastructure: turning "it's slow" into "here's why"

This part took about as much time as the gateway itself, and it's the part most demo projects skip entirely: Prometheus, Grafana, Tempo, Loki, and Alloy, wired up so you can follow one request all the way through.

The four pieces carry genuinely different signals, and it's easy to blur them together if you haven't set one up before. Prometheus is metrics, and it's pull-based — it scrapes /actuator/prometheus on all four Spring services every 15 seconds, nothing pushes to it, it just polls and stores time series: request counts, latencies, JVM stats, the gateway.rate_limit.requests counter from earlier, Resilience4j's breaker state. Tempo is traces, and it's push-based — each service's Micrometer Tracing setup (the OpenTelemetry bridge) ships spans to Tempo's OTLP endpoint as requests happen, which is what lets you open one request and watch it hop gateway → api-server → product-service with timing for each leg. Loki is logs, but on its own it collects nothing — it's just a store with a query API, and something else has to actually ship logs into it. That something is Alloy: it mounts the Docker socket, discovers every container on the network automatically, tails stdout/stderr, and pulls trace_id, span_id, and level out of the structured JSON logs. The IDs go in as structured metadata rather than labels (a per-request value as a Loki label would wreck its index cardinality), while level becomes a proper label you can filter on.

Getting structured JSON logs with those fields out of Spring Boot, by the way, is one property, not a custom log appender:

logging:
  structured:
    format:
      console: logstash

The whole point is the trace_id running through all three systems. The ID a service stamps on its log line is the same ID Tempo has on the matching span, so Grafana can jump straight from a log line to the full trace, or from a span back to the logs around it. Metrics tell you something is wrong, traces show you where, logs tell you why — and you get all three without three separate manual lookups.

The bugs the observability stack itself found

This is the part I didn't see coming. Building a smoke-test suite that actually exercises this stack end-to-end turned up two real bugs that no unit test would ever have caught, because both of them only exist once the whole system is running together.

Bug one: a rate-limit bucket that could never fire. The per-IP limit was originally set above every route limit — 15/sec burst 30, versus 10/sec burst 20 for reads. Since the route bucket is shared globally per route no matter who's calling, it always rejected first. The IP-limit code path was live, deployed, and completely unreachable — nothing was ever going to make it past the route bucket to even reach the IP check. The fix wasn't code at all, just retuning application.yml so the IP bucket actually sits between the client bucket and the route buckets:

app:
  rate-limit:
    route:
      read:  { replenish-rate: 10, burst-capacity: 20 }
      write: { replenish-rate: 2,  burst-capacity: 5 }
    client:  { replenish-rate: 5,  burst-capacity: 10 }
    ip:      { replenish-rate: 8,  burst-capacity: 15 }   # was 15/30 — sat above every route bucket

Bug two: a trusted-proxy check that only understood loopback. The IP resolver only matched exact addresses — 127.0.0.1, ::1. Fine if the service runs directly on your laptop. Not fine in Docker: a Dockerized gateway never sees loopback as the peer address for host-originated traffic, because Docker's userland proxy rewrites the source to the bridge network's gateway IP on the way in. So X-Forwarded-For was quietly ignored the moment this ran in Compose, and every simulated client IP collapsed into one bucket. Spring Security has IpAddressMatcher for this exact situation, but it implements the servlet-based RequestMatcher interface, which isn't even on the classpath in a WebFlux-only module — so I wrote a small CIDR matcher instead:

private record CidrRange(byte[] networkBytes, int prefixLength) {

    boolean contains(String candidateIp) {
        byte[] candidateBytes = addressBytes(candidateIp);
        if (candidateBytes.length != networkBytes.length) {
            return false;
        }
        int fullBytes = prefixLength / 8;
        for (int i = 0; i < fullBytes; i++) {
            if (networkBytes[i] != candidateBytes[i]) {
                return false;
            }
        }
        int remainingBits = prefixLength % 8;
        if (remainingBits == 0) {
            return true;
        }
        int mask = 0xFF << (8 - remainingBits) & 0xFF;
        return (networkBytes[fullBytes] & mask) == (candidateBytes[fullBytes] & mask);
    }
}

...and pinned the Compose network's subnet so there was an actual, stable CIDR to trust instead of whatever address Docker happened to hand out that day:

# docker-compose.infra.yml
networks:
  gateway-sample-net:
    ipam:
      config:
        - subnet: 172.28.1.0/24

# application-local.yml
app:
  rate-limit:
    trusted-proxies:
      - 127.0.0.1
      - "0:0:0:0:0:0:0:1"
      - 172.28.1.0/24   # the Compose bridge network, not loopback

Neither of these would ever show up in an isolated unit test — one's a cross-component config ordering problem, the other only exists once you're actually inside Docker's networking. Which is basically the whole argument for building the observability and end-to-end tests alongside the feature, not after it: some bugs genuinely only live at the seams between components, and a unit test by definition can't see a seam.

What's deliberately not here

No Lua-script rate limiting, for the reasons above. No ops dashboard in the Vue UI yet — that's still on the list. It's a phased build, and I'd rather tell you what's missing than let you assume the whole thing is further along than it is.

Want to poke at it yourself:

cp .env.example .env
docker compose -f docker-compose.infra.yml -f docker-compose.yml up -d --build

DEMO.md walks through triggering each failure mode by hand.

Curious what other people think about the rate-limiter tradeoff specifically — Lua script from day one, or is sequential-buckets-with-refund good enough for something at this scale? Where do you draw that line in your own systems?