How to Find Which Proxy Returned a 502 or 504 — A Reproducible NGINX Debugging Lab

A 502 Bad Gateway or 504 Gateway Timeout tells you that something failed around a gateway boundary.

It does not necessarily tell you which gateway failed.

That distinction matters when a request crosses multiple layers:

Client
  |
  v
Edge proxy
  |
  v
API gateway
  |
  v
Application

If the client receives:

504 Gateway Timeout

several different things may have happened:

the gateway could not establish a connection to its upstream
the gateway connected successfully but waited too long for a response
an outer proxy could have timed out while an inner gateway was still waiting
the application could have continued processing after an upstream proxy had already abandoned the request

I built a small Docker lab with two NGINX proxies and a Python application to make these failure modes deterministic.

The most interesting result was one request for which the different layers recorded:

External client:  504
Edge NGINX:       504
Gateway NGINX:    499
Application:      eventually attempts 200

Those results are not contradictory.

They describe the same request from different points in the request path.

This lab shows how to reconstruct that path.

What we're building

The topology is deliberately simple:

                     localhost:8080
Client  ───────────► Edge NGINX
                         |
                         v
                    Gateway NGINX
                         |
                         v
                     Python API

For debugging, each layer is also exposed directly:

Application    localhost:8000
Gateway        localhost:18081
Edge           localhost:8080

I ran the lab on macOS using Colima with Docker Engine 29.2.1 on ARM64.

The containers used:

NGINX     1.31.4
Python    3.13.15

Container IP addresses shown later are specific to my run and should not be treated as fixed values.

Project structure

Create:

proxy-debug-lab/
├── app/
│   ├── app.py
│   └── Dockerfile
├── compose.yaml
├── edge/
│   └── nginx.conf
└── gateway/
    └── nginx.conf

Build a small backend that can be intentionally slow

The backend uses only Python's standard library.

app/app.py:

import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        parsed = urlparse(self.path)
        trace_id = self.headers.get("X-Request-ID", "-")

        if parsed.path == "/health":
            self.respond(
                200,
                {
                    "status": "healthy",
                    "trace_id": trace_id,
                },
            )
            return

        if parsed.path == "/fast":
            self.respond(
                200,
                {
                    "status": "ok",
                    "endpoint": "fast",
                    "trace_id": trace_id,
                },
            )
            return

        if parsed.path == "/slow":
            params = parse_qs(parsed.query)

            try:
                seconds = float(params.get("seconds", ["5"])[0])
            except ValueError:
                seconds = 5

            seconds = max(0, min(seconds, 30))

            print(
                f"layer=app trace={trace_id} "
                f"path={parsed.path} sleeping={seconds}s",
                flush=True,
            )

            time.sleep(seconds)

            self.respond(
                200,
                {
                    "status": "ok",
                    "endpoint": "slow",
                    "slept_seconds": seconds,
                    "trace_id": trace_id,
                },
            )
            return

        self.respond(
            404,
            {
                "status": "not_found",
                "path": parsed.path,
                "trace_id": trace_id,
            },
        )

    def respond(self, status, payload):
        body = (json.dumps(payload) + "\n").encode()

        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()

        try:
            self.wfile.write(body)
        except BrokenPipeError:
            print("layer=app client_connection_closed=true", flush=True)

    def log_message(self, fmt, *args):
        trace_id = self.headers.get("X-Request-ID", "-")

        print(
            f'layer=app trace={trace_id} message="{fmt % args}"',
            flush=True,
        )


if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), Handler)

    print("layer=app listening=0.0.0.0:8000", flush=True)

    server.serve_forever()

The endpoint we care about most is:

/slow?seconds=5

It accepts the request, sleeps for five seconds without sending response headers, and then attempts to return 200.

That lets us deliberately place shorter proxy timeouts in front of it.

The Dockerfile is minimal:

FROM python:3.13.15-alpine3.24

WORKDIR /app

COPY app.py .

EXPOSE 8000

CMD ["python", "-u", "app.py"]

Instrument the gateway

Before introducing failures, make the proxies observable.

gateway/nginx.conf:

events {}

http {
    map $http_x_request_id $trace_id {
        default $http_x_request_id;
        ""      $request_id;
    }

    log_format trace
        '$time_iso8601 '
        'layer=gateway '
        'trace=$trace_id '
        'status=$status '
        'upstream_status=$upstream_status '
        'request_time=$request_time '
        'upstream_response_time=$upstream_response_time '
        'upstream_addr=$upstream_addr '
        'request="$request"';

    access_log /var/log/nginx/access.log trace;
    error_log  /var/log/nginx/error.log notice;

    server {
        listen 80;

        add_header X-Gateway-Observed "true" always;

        location = /health {
            proxy_pass http://app:8000/health;

            proxy_set_header X-Request-ID $trace_id;
            proxy_read_timeout 2s;
        }

        location = /fast {
            proxy_pass http://app:8000/fast;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        # Reachable host, but nothing listens on port 9999.
        location = /bad-gateway {
            proxy_pass http://app:9999;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 2s;
        }

        # App sleeps for 5s, but Gateway waits only 2s.
        location = /gateway-timeout {
            proxy_pass http://app:8000/slow?seconds=5;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 2s;
        }

        # App sleeps for 5s and Gateway is willing to wait 10s.
        location = /edge-timeout {
            proxy_pass http://app:8000/slow?seconds=5;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        error_page 502 = @gateway_502;
        error_page 504 = @gateway_504;

        location @gateway_502 {
            default_type text/plain;

            add_header X-Gateway-Observed "true" always;
            add_header X-Debug-Generated-By "gateway" always;

            return 502 "gateway generated 502\n";
        }

        location @gateway_504 {
            default_type text/plain;

            add_header X-Gateway-Observed "true" always;
            add_header X-Debug-Generated-By "gateway" always;

            return 504 "gateway generated 504\n";
        }
    }
}

The important log fields are:

$status
$upstream_status
$request_time
$upstream_response_time
$upstream_addr

NGINX also exposes fields such as $upstream_connect_time and $upstream_header_time, which are useful additions in a production logging format.

Configure the outer edge proxy

edge/nginx.conf:

events {}

http {
    map $http_x_request_id $trace_id {
        default $http_x_request_id;
        ""      $request_id;
    }

    log_format trace
        '$time_iso8601 '
        'layer=edge '
        'trace=$trace_id '
        'status=$status '
        'upstream_status=$upstream_status '
        'request_time=$request_time '
        'upstream_response_time=$upstream_response_time '
        'upstream_addr=$upstream_addr '
        'request="$request"';

    access_log /var/log/nginx/access.log trace;
    error_log  /var/log/nginx/error.log notice;

    server {
        listen 80;

        add_header X-Edge-Observed "true" always;
        add_header X-Request-ID $trace_id always;

        # If Gateway itself returns an error, pass that response through.
        proxy_intercept_errors off;

        location = /health {
            proxy_pass http://gateway/health;

            proxy_set_header X-Request-ID $trace_id;
            proxy_read_timeout 3s;
        }

        location = /fast {
            proxy_pass http://gateway/fast;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        location = /bad-gateway {
            proxy_pass http://gateway/bad-gateway;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 5s;
        }

        # Give Gateway enough time to generate its own 504.
        location = /gateway-timeout {
            proxy_pass http://gateway/gateway-timeout;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 10s;
        }

        # Edge gives up first.
        location = /edge-timeout {
            proxy_pass http://gateway/edge-timeout;

            proxy_set_header X-Request-ID $trace_id;
            proxy_connect_timeout 2s;
            proxy_read_timeout 2s;
        }

        error_page 502 = @edge_502;
        error_page 504 = @edge_504;

        location @edge_502 {
            default_type text/plain;

            add_header X-Edge-Observed "true" always;
            add_header X-Request-ID $trace_id always;
            add_header X-Debug-Generated-By "edge" always;

            return 502 "edge generated 502\n";
        }

        location @edge_504 {
            default_type text/plain;

            add_header X-Edge-Observed "true" always;
            add_header X-Request-ID $trace_id always;
            add_header X-Debug-Generated-By "edge" always;

            return 504 "edge generated 504\n";
        }
    }
}

The X-Debug-Generated-By headers are lab instrumentation, not standard NGINX headers.

I added them so that the experiments make the response origin visually obvious.

In a real environment, correlation IDs and logs are much more important.

Run everything with Docker Compose

compose.yaml:

services:
  app:
    build:
      context: ./app

    ports:
      - "8000:8000"

  gateway:
    image: nginx:1.31.4-alpine3.24

    volumes:
      - ./gateway/nginx.conf:/etc/nginx/nginx.conf:ro

    ports:
      - "18081:80"

    depends_on:
      - app

  edge:
    image: nginx:1.31.4-alpine3.24

    volumes:
      - ./edge/nginx.conf:/etc/nginx/nginx.conf:ro

    ports:
      - "8080:80"

    depends_on:
      - gateway

Start the environment:

docker compose up -d --build

Then check:

docker compose ps

Validate the loaded NGINX configurations:

docker compose exec gateway nginx -t
docker compose exec edge nginx -t

Both should end with:

syntax is ok
test is successful

Establish a healthy baseline first

Always prove the healthy path before deliberately breaking it.

Test the application:

curl -i http://localhost:8000/health

Then Gateway → Application:

curl -i http://localhost:18081/health

Then the complete path:

curl -i http://localhost:8080/health

The complete request returned:

HTTP/1.1 200 OK
X-Gateway-Observed: true
X-Edge-Observed: true
X-Request-ID: d66701d1ec8700e4bf111ab99b49372d

I then measured a normal /fast request:

curl -sS -D - -o /dev/null \
  -w '\nstatus=%{http_code}\nconnect=%{time_connect}s\nttfb=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  http://localhost:8080/fast

One saved run produced:

status=200
connect=0.000248s
ttfb=0.010474s
total=0.010642s

Both proxies recorded successful upstream responses.

Gateway:

layer=gateway
status=200
upstream_status=200

Edge:

layer=edge
status=200
upstream_status=200

Now we have a control case.

Failure 1: Immediate upstream connection refusal

The /bad-gateway route points Gateway at:

app:9999

The application container exists and is reachable, but nothing is listening on that port.

Request it through the full chain:

curl -sS -D - http://localhost:8080/bad-gateway

The response was:

HTTP/1.1 502 Bad Gateway
X-Gateway-Observed: true
X-Debug-Generated-By: gateway
X-Edge-Observed: true

gateway generated 502

The measured request:

status=502
connect=0.000342s
ttfb=0.001889s
total=0.002027s

Gateway's error log explained why:

connect() failed (111: Connection refused) while connecting to upstream

Its access log showed:

layer=gateway
status=502
upstream_status=502
upstream_addr=172.20.0.2:9999

Edge recorded:

layer=edge
status=502
upstream_status=502

The sequence is therefore:

Client
  |
  v
Edge              reached successfully
  |
  v
Gateway           reached successfully
  |
  X
app:9999          connection refused

Gateway generates 502
Edge forwards 502

The client status alone does not show that sequence.

The adjacent logs do.

A backend being "down" did not always produce a 502

I also tried a less controlled experiment:

docker compose stop app

I initially expected another 502.

That did not happen in this environment.

The gateway attempted to connect to the application's previous container address, but the connection attempt did not immediately fail.

Instead, it waited until:

proxy_connect_timeout 2s;

expired.

The client received:

HTTP/1.1 504 Gateway Time-out
X-Debug-Generated-By: gateway

with:

status=504
connect=0.000293s
ttfb=2.003997s
total=2.004799s

Gateway logged:

upstream timed out ... while connecting to upstream

Compare that with the closed-port experiment:

Immediate connection refusal
    -> 502 in this lab

Connection attempt times out
    -> 504 in this lab

I would not generalize this into:

stopped container = 504

The network behavior of a stopped workload can vary by environment.

The useful lesson is:

"The backend is down" is not enough information to predict the status code. You need to determine how the upstream connection failed.

After this experiment I restored the application:

docker compose start app

Failure 2: Gateway connects successfully but times out reading the response

Now we create a very different failure.

The application waits:

5 seconds

Gateway waits:

2 seconds

Edge waits:

10 seconds

So Gateway must be the first layer to give up.

The route is:

location = /gateway-timeout {
    proxy_pass http://app:8000/slow?seconds=5;

    proxy_set_header X-Request-ID $trace_id;
    proxy_connect_timeout 2s;
    proxy_read_timeout 2s;
}

Request:

curl -sS -D - \
  -o /dev/null \
  -w '\nstatus=%{http_code}\nttfb=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  http://localhost:8080/gateway-timeout

The result:

HTTP/1.1 504 Gateway Time-out
X-Gateway-Observed: true
X-Debug-Generated-By: gateway
X-Edge-Observed: true

status=504
ttfb=2.005843s
total=2.005997s

A saved run measured:

status=504
connect=0.000261s
ttfb=2.004355s
total=2.005004s

The application proves that the request successfully reached it:

layer=app
path=/slow
sleeping=5.0s

Gateway logged:

upstream timed out (110: Operation timed out)
while reading response header from upstream

This is different from the previous connect-timeout error:

while connecting to upstream

So we have now produced two client-visible 504 responses during different phases:

Connection phase
    -> timeout while connecting to upstream

Response phase
    -> timeout while reading response header

According to the NGINX proxy module documentation, proxy_connect_timeout controls the timeout for establishing the upstream connection, while proxy_read_timeout controls the interval between successive read operations from the upstream.

Those are different failure boundaries.

Why the application later logged 200

There was another interesting result.

The application continued sleeping after Gateway had already returned the 504.

Eventually it reached:

"GET /slow?seconds=5 HTTP/1.1" 200

and then:

layer=app client_connection_closed=true

That application log entry does not mean the original client received 200 OK.

Our Python handler logs the intended response status as it starts generating the response.

By that point, Gateway had already timed out and closed the upstream connection.

When Python later tried writing the body, the socket was gone and the handler caught BrokenPipeError.

The client had already received:

504 Gateway Time-out

This is a useful reminder that an application-side status log and a client-visible status are observations at different boundaries.

Failure 3: Same client-visible 504, different proxy

Now reverse the timeout relationship:

Application delay       5s
Gateway read timeout   10s
Edge read timeout       2s

Gateway is willing to wait long enough for the application.

Edge is not.

Gateway:

location = /edge-timeout {
    proxy_pass http://app:8000/slow?seconds=5;

    proxy_set_header X-Request-ID $trace_id;
    proxy_connect_timeout 2s;
    proxy_read_timeout 10s;
}

Edge:

location = /edge-timeout {
    proxy_pass http://gateway/edge-timeout;

    proxy_set_header X-Request-ID $trace_id;
    proxy_connect_timeout 2s;
    proxy_read_timeout 2s;
}

Run:

curl -sS -D - \
  -o /dev/null \
  -w '\nstatus=%{http_code}\nttfb=%{time_starttransfer}s\ntotal=%{time_total}s\n' \
  http://localhost:8080/edge-timeout

The client again saw:

HTTP/1.1 504 Gateway Time-out

But now the diagnostic header changed:

X-Debug-Generated-By: edge

The measured request:

status=504
connect=0.000274s
ttfb=2.005233s
total=2.005678s

From the client's point of view, this looked almost identical to the previous gateway timeout.

Internally, it was completely different.

The 499 that reveals the real sequence

This was the most useful part of the experiment.

The request ID was:

d9921f9d15f2df8ea396c7baf6c81d5a

I correlated it across the three services:

docker compose logs --no-color app gateway edge \
  | grep 'd9921f9d15f2df8ea396c7baf6c81d5a'

The application recorded:

layer=app
trace=d9921f9d15f2df8ea396c7baf6c81d5a
path=/slow
sleeping=5.0s

Edge recorded:

layer=edge
trace=d9921f9d15f2df8ea396c7baf6c81d5a
status=504
request_time=2.003
upstream_response_time=2.003
request="GET /edge-timeout HTTP/1.1"

Gateway recorded:

layer=gateway
trace=d9921f9d15f2df8ea396c7baf6c81d5a
status=499
upstream_status=-
request_time=2.003
upstream_response_time=2.003
request="GET /edge-timeout HTTP/1.1"

Edge's error log also said:

upstream timed out ... while reading response header from upstream

The sequence was:

                        app still processing
                               |
                               |
Client -> Edge -> Gateway -> Application
           |
           | 2 second timeout expires
           |
           +--> Edge returns 504 to Client
           |
           +--> Edge closes its request to Gateway

From Gateway's point of view, Edge is the client.

Gateway was willing to keep waiting for the application, but its client disconnected first.

NGINX therefore recorded 499.

NGINX defines 499 internally as NGX_HTTP_CLIENT_CLOSED_REQUEST. It is not a standard HTTP status sent by the external client; it is used by NGINX to represent the client closing the request before a response could be completed.

The application continued its five-second operation and later attempted to send its 200.

By then, the original request path was already gone.

So one request resulted in:

External client     504
Edge                504
Gateway             499
Application         eventually attempts 200

All four observations are valid at their respective boundaries.

Be careful interpreting `$upstream_status` alone

There was one more subtle result.

For the edge-generated timeout, Edge logged:

status=504
upstream_status=504

It would be tempting to conclude:

Gateway returned HTTP 504 to Edge.

But Gateway's own log for the same trace ID showed:

status=499
upstream_status=-

and Edge's error log explicitly said that Edge itself timed out while reading from its upstream.

So in this experiment, using Edge's $upstream_status alone to identify which process emitted the client-visible 504 would have produced the wrong conclusion.

Correlation across adjacent layers gave us the actual sequence.

This is one reason I prefer to combine:

local status
upstream status
request timing
upstream timing
error log
correlation ID

instead of treating any single field as the whole story.

The NGINX upstream module documentation describes the available upstream variables, including $upstream_status, $upstream_connect_time, $upstream_header_time, and $upstream_response_time.

A practical debugging sequence

When a request crosses several gateways or proxies, I would debug it in this order:

Propagate one correlation ID through every hop.
At every proxy, log the local response status and upstream information.
Record timing for different phases of the request. For NGINX, useful fields include:

   $request_time
   $upstream_connect_time
   $upstream_header_time
   $upstream_response_time

Read the error message carefully.

These are materially different:

   while connecting to upstream

and:

   while reading response header from upstream

Find the same request ID at the adjacent layer before changing timeout values.
Determine which component actually generated the client-visible error.
Only then investigate why the upstream was unavailable or slow.

A timeout setting is often where a problem becomes visible.

It is not necessarily where the underlying problem originated.

Results

The deterministic experiments produced:

Scenario	Client result	Response generated by	Observed time
Healthy request	`200`	Application response propagated	~10 ms
Reachable host, closed upstream port	`502`	Gateway	~2 ms
Slow application, Gateway waits 2s	`504`	Gateway	~2.005 s
Slow application, Gateway waits 10s, Edge waits 2s	`504`	Edge	~2.006 s

The additional stopped-container experiment produced another Gateway 504, but because the connection attempt timed out, not because Gateway timed out waiting for application response data.

So the model I would avoid is:

502 = backend down
504 = backend slow

A more useful set of questions is:

Which hop failed?

During which phase did it fail?

Which component generated the response?

What did the adjacent hop record for the same request?

Once those questions are answered, a multi-proxy 502 or 504 becomes much easier to localize.

References

Disclosure

AI assistance was used for technical research and editorial review while developing this article. I manually built and ran the lab, and the configurations, commands, response codes, timings, request traces, and log behavior described above were validated against the running environment before publication.

How to Find Which Proxy Returned a 502 or 504 — A Reproducible NGINX Debugging Lab

What we're building

Project structure

Build a small backend that can be intentionally slow

Instrument the gateway

Configure the outer edge proxy

Run everything with Docker Compose

Establish a healthy baseline first

Failure 1: Immediate upstream connection refusal

A backend being "down" did not always produce a 502

Failure 2: Gateway connects successfully but times out reading the response

Why the application later logged 200

Failure 3: Same client-visible 504, different proxy

The 499 that reveals the real sequence

Be careful interpreting `$upstream_status` alone

A practical debugging sequence

Results

References

Disclosure

Tags

Author

Stats

Published

You Might Also Like

Git Said It Succeeded. The State Said Otherwise.

My Commit Said "Stop Trusting the Receipt." That Is the Commit Where the Trusting Started.

Your Agent Planned the Right Tools. It Still Crashed the Machine.

Half the Requests Wearing ChatGPT's Name Were Scanning Us for Secrets

Your App Center List Is Not a Security Audit

Running Coding Agents in Parallel with Git Worktrees

How to Find Which Proxy Returned a 502 or 504 — A Reproducible NGINX Debugging Lab

What we're building

Project structure

Build a small backend that can be intentionally slow

Instrument the gateway

Configure the outer edge proxy

Run everything with Docker Compose

Establish a healthy baseline first

Failure 1: Immediate upstream connection refusal

A backend being "down" did not always produce a 502

Failure 2: Gateway connects successfully but times out reading the response

Why the application later logged 200

Failure 3: Same client-visible 504, different proxy

The 499 that reveals the real sequence

Be careful interpreting $upstream_status alone

A practical debugging sequence

Results

References

Disclosure

Tags

Author

Stats

Published

You Might Also Like

Git Said It Succeeded. The State Said Otherwise.

My Commit Said "Stop Trusting the Receipt." That Is the Commit Where the Trusting Started.

Your Agent Planned the Right Tools. It Still Crashed the Machine.

Half the Requests Wearing ChatGPT's Name Were Scanning Us for Secrets

Your App Center List Is Not a Security Audit

Running Coding Agents in Parallel with Git Worktrees

Be careful interpreting `$upstream_status` alone