502 Bad Gateway From Nginx: Which Side Is Actually Broken

The short answer

502 means nginx reached your application and got something it could not use. 504 means nginx gave up waiting. 503 usually means no upstream was available at all.

The error log has the specific reason:

tail -f /var/log/nginx/error.log

2025/11/25 09:14:22 [error] 1234#0: *5678 connect() failed (111: Connection refused)
while connecting to upstream, client: 10.0.0.5, server: api.example.com,
request: "GET /orders HTTP/1.1", upstream: "http://127.0.0.1:8080/orders"

The parenthesised errno is the diagnosis:

Tested on nginx 1.27.

The six causes

1. Connection refused: nothing is listening

The most common and the simplest.

ss -lntp | grep 8080
curl -v http://127.0.0.1:8080/health

If nothing is listening, your application is not running or it crashed on startup. Check its own logs, not nginx's.

The variant that catches people: the app is listening on 127.0.0.1 and nginx is configured to reach it on the container or host IP, or the reverse. In a container, an app bound to 127.0.0.1 is unreachable from outside the container entirely.

app.listen(8080, "0.0.0.0");     // reachable
app.listen(8080, "127.0.0.1");   // localhost only

2. Upstream prematurely closed connection

Your application accepted the request, started responding, and then died or closed the socket.

upstream prematurely closed connection while reading response header from upstream

Causes: the process crashed mid-request, a worker was killed by an OOM killer, a request timeout inside the app framework, or a worker recycling policy that killed a busy worker.

Check on the application side:

dmesg -T | grep -i "killed process"        # OOM killer
journalctl -u myapp --since "10 min ago"

If it is the OOM killer, the memory limit rather than nginx is your problem, and the 502 is a symptom two layers away from the cause.

For gunicorn and uwsgi specifically, check the worker timeout. A worker killed for exceeding it produces exactly this, and the fix is either raising the timeout or fixing the slow handler.

3. Timeouts: this is actually a 504

upstream timed out (110: Connection timed out) while reading response header

nginx waited and gave up. The defaults are 60 seconds each for connect, send, and read.

location /api/ {
    proxy_pass http://backend;
    proxy_connect_timeout 5s;
    proxy_send_timeout 30s;
    proxy_read_timeout 30s;
}

Raising these is the instinct and usually the wrong move. If a request takes over 60 seconds, the fix is in the application. Raising the timeout means holding connections and workers longer, which reduces capacity and makes an overload worse.

The exception is genuinely long operations like report generation or file processing. Those should be asynchronous with a job id and a polling endpoint, not a 5 minute HTTP request. Timeouts should get shorter as you go deeper, and a long one at the edge is a design smell.

4. No live upstreams

no live upstreams while connecting to upstream

nginx has marked every backend as failed and is not trying any of them.

This is passive health checking. After max_fails failures within fail_timeout, a server is removed from rotation for fail_timeout seconds.

upstream backend {
    server 10.0.0.10:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
}

The trap: with default settings, a brief blip across all backends takes everything out of rotation for 30 seconds, and during that window nginx returns 502 without even attempting a connection. Your backends recover, nginx does not notice until the timer expires, and the outage is longer than the actual failure.

For a small number of backends, consider max_fails=0 on at least one server so it is never removed, or use active health checks if you have nginx Plus or a different proxy.

5. Header too large

upstream sent too big header while reading response header from upstream

Your application returned headers exceeding nginx's buffer. Usually a very large cookie, a long Set-Cookie chain, or a verbose error header.

proxy_buffer_size 16k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 32k;

Worth asking why the headers are that large first. A session cookie of several kilobytes is a design problem that also costs you on every request.

6. Permission denied on a Unix socket

connect() to unix:/run/app.sock failed (13: Permission denied)

nginx runs as www-data or nginx and the socket is owned by someone else, or SELinux is blocking it.

ls -l /run/app.sock
sudo -u www-data curl --unix-socket /run/app.sock http://localhost/health

For SELinux:

sudo ausearch -m avc -ts recent
sudo setsebool -P httpd_can_network_connect 1

The httpd_can_network_connect boolean catches a lot of people on RHEL derived distributions, where nginx is blocked from making outbound connections by default and every proxy_pass fails with permission denied.

Getting better diagnostics

Log the upstream details. The default access log does not include them:

log_format upstream '$remote_addr - $status "$request" '
                    'ut=$upstream_response_time urt=$upstream_connect_time '
                    'us=$upstream_status ua=$upstream_addr '
                    'rt=$request_time';

access_log /var/log/nginx/access.log upstream;

$upstream_response_time versus $request_time is the key comparison. If they are close, your application is slow. If $request_time is much larger, the time is in nginx, the network, or sending to a slow client.

$upstream_addr showing multiple addresses means nginx retried against several backends, which tells you it is not one bad instance.

Raise the log level temporarily:

error_log /var/log/nginx/error.log debug;

Very verbose. Useful for ten minutes on a reproducing case, not permanently.

Correlate with a request id. Generate one at nginx and pass it through:

proxy_set_header X-Request-Id $request_id;

Log it on both sides. Now a 502 in the nginx log can be matched against the exact application log lines, which is the difference between guessing and knowing.

Retries and idempotency

nginx retries failed upstream requests by default:

proxy_next_upstream error timeout http_502 http_503;

That includes non-idempotent requests unless you say otherwise. A POST that timed out after the backend processed it will be retried against another backend, and now the order is placed twice.

proxy_next_upstream error timeout non_idempotent;   # explicit opt-in, be careful

The safe configuration is to not retry POST at all and to handle idempotency at the application layer with an idempotency key, which you need anyway for client retries.

A diagnostic order

1. Read the error log line and the errno
2. curl the upstream directly from the nginx host, bypassing nginx
3. If that works, the problem is in nginx configuration or networking between them
4. If it fails, the problem is the application, and nginx is just reporting it
5. Compare $upstream_response_time and $request_time in the access log
6. Check for OOM kills and worker restarts on the application side
7. Check whether all upstreams got marked down and how long ago