Nginx Reverse Proxy 502 / 504 Troubleshooting

Q: What's the fastest first step when Nginx returns 502?

Check whether the upstream is actually listening on the port Nginx is proxying to: `ss -lnt | grep :8080`. If nothing is listening, start or restart the upstream service. If it is listening, look at upstream logs and Nginx `error.log`.

Q: What's the fastest first step when Nginx returns 504?

504 means the upstream exceeded `proxy_read_timeout` (default 60s). Check `journalctl -u nginx -f` for a "upstream timed out" line — the exact upstream address is printed there. Then either fix the slow upstream or bump the timeout deliberately.

Q: When should I raise proxy_read_timeout?

Only when the upstream legitimately needs that long (report generation, ML inference, batch jobs). It's a symptom mask, not a fix. Bump in steps (60 → 120 → 300) and always log the reason in a comment so the next engineer isn't surprised.

Q: How do I distinguish 502 vs 504 from Nginx logs?

`error.log` will say "connect() failed" or "upstream prematurely closed connection" for 502, and "upstream timed out (110: Operation timed out)" for 504. The exact string tells you which class of problem you have.

Q: Why does my HTTPS reverse proxy still return 502 after I enabled TLS?

Because `proxy_pass` is still talking HTTP to an HTTPS backend. Match the protocol: `proxy_pass https://backend:443;` plus `proxy_ssl_server_name on;` and `proxy_ssl_name` set to the backend's TLS SNI hostname.

The 30-Second Diagnosis

When Nginx shows you 502 Bad Gateway or 504 Gateway Timeout, it has already talked to the upstream (or tried to) and given up. The upstream is either not reachable, not speaking the protocol Nginx expects, or too slow. Work in this order:

Is the upstream actually listening? → ss -lnt | grep :PORT
Is the upstream seeing the request? → upstream logs
Does proxy_pass point at the right thing? → reload Nginx config in error.log
Is TLS between Nginx and upstream misconfigured? → check proxy_pass protocol and proxy_ssl_*
Is the upstream just slow? → check proxy_read_timeout and upstream processing time

If you get past step 1 and still see 502, jump to the 502 Decision Tree. If the error is 504, the upstream is reachable but slow — 504 Timeout Decision Tree will walk through it.

5 Real Failure Scenarios

1. Upstream Not Running

The most common 502 cause, and the one that trips up the freshest engineers.

Symptom: 502 immediately after deploy, Nginx error.log says:

connect() failed (111: Connection refused) while connecting to upstream
upstream: "http://127.0.0.1:8080/", host: "127.0.0.1:8080"

Diagnosis:

ss -lnt | grep :8080    # Empty means nobody listening
systemctl status myapp   # Is the upstream service up?

Fix: Restart the upstream. Check for boot-time failures (journalctl -u myapp -b).

2. Wrong Address Behind `proxy_pass`

A typo, a missing port, a wrong protocol — all produce the same 502.

Common culprits: - proxy_pass http://127.0.0.1; → defaults to port 80, not 8080. Fix: proxy_pass http://127.0.0.1:8080; - proxy_pass http://backend:8080; but backend doesn't resolve. Fix: add to /etc/hosts or use Docker DNS (extra_hosts:). - proxy_pass https://backend; but backend only speaks HTTP. Fix: change to http://.

Diagnosis:

grep -n 'proxy_pass' /etc/nginx/sites-enabled/default
dig backend        # Does the hostname resolve?
curl -v http://backend:8080/health   # From the Nginx host, does the upstream respond?

3. TLS Mismatch Between Nginx and Upstream

If you flipped the upstream to HTTPS and forgot to update proxy_pass, Nginx will still try HTTP and get a garbled response.

Symptom: Nginx log shows upstream sent invalid header or upstream sent no valid HTTP/1.0 header while reading response header from upstream.

Diagnosis:

# Nginx config shows proxy_pass http://backend:8443 (HTTP to HTTPS port)
# Test from the Nginx host:
curl -vk https://backend:8443/     # Should work
curl -v http://backend:8443/       # Garbled response confirms the mismatch

Fix:

proxy_pass https://backend:8443;
proxy_ssl_server_name on;
proxy_ssl_name backend.example.com;   # SNI hostname
proxy_ssl_protocols TLSv1.2 TLSv1.3;
proxy_ssl_verify off;                 # For self-signed cert; set on + CA if you own the cert

4. Keepalive Stale Connection

Long-lived upstreams (Java, Go net/http, Rails) tend to close idle connections from the load balancer. Nginx may hand a stale socket to a real request.

Symptom: intermittent 502s, only on the first request after a quiet period, upstream logs show "connection closed while reading headers" or similar.

Diagnosis:

# Check upstream keepalive settings in Nginx
grep -n 'keepalive' /etc/nginx/conf.d/upstream.conf

# On the upstream, look for the connection lifetime
ss -tnp | grep ':8080' | awk '{print $4}'   # Look for CLOSE_WAIT on many sockets

Fix:

upstream backend {
    server 127.0.0.1:8080;
    keepalive 32;              # Max idle conns Nginx keeps open to upstream
    keepalive_timeout 60s;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;          # Required for keepalive upstreams
        proxy_set_header Connection "";  # Empty Connection header for keepalive
    }
}

5. Backend Protocol Mismatch (HTTP/1.1 vs HTTP/2)

If your upstream requires HTTP/2 (gRPC, some modern Java servers), the default HTTP/1.1 from Nginx will cause weird failures.

Symptom: upstream sent invalid header or gRPC-specific errors like http2 framing layer.

Fix:

location / {
    proxy_pass http://backend:50051;
    grpc_pass grpc://backend:50051;     # For gRPC
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

20-Command Cheat Sheet

Copy-paste to diagnose 502/504 quickly.

# 1. What does the user see?
curl -Iv https://your-domain.com/health

# 2. What's Nginx logging right now?
sudo journalctl -u nginx -f

# 3. What's the upstream logging?
sudo journalctl -u myapp -f

# 4. Is the upstream listening?
ss -lnt | grep :8080

# 5. Can Nginx reach the upstream?
sudo -u nginx curl -v http://127.0.0.1:8080/health

# 6. Which config is actually loaded?
sudo nginx -T 2>&1 | grep -A5 'proxy_pass'

# 7. Config syntax OK?
sudo nginx -t

# 8. Reload config
sudo systemctl reload nginx

# 9. What's the connection state between Nginx and upstream?
ss -tnp | grep ':8080'

# 10. Upstream service status
systemctl status myapp
systemctl status myapp.service --full

# 11. Upstream CPU/memory (spikes cause slowness → 504)
top -p $(pgrep -f myapp | head -1)

# 12. Nginx worker count and state
sudo nginx -T | grep worker_connections
ps aux | grep nginx

# 13. Check the exact Nginx error message
grep -E '502|504|timed out|refused' /var/log/nginx/error.log | tail -20

# 14. Check upstream response time
curl -w '@format.txt' -o /dev/null https://your-domain.com/
# format.txt: time_total, time_connect, time_starttransfer

# 15. Check upstream TLS
openssl s_client -connect backend:8443 -servername backend.example.com

# 16. Check upstream via HTTP (if you suspect TLS mismatch)
curl -v http://backend:8443/   # Garbled = mismatch

# 17. Nginx error.log level
grep 'error_log' /etc/nginx/nginx.conf

# 18. Request body size limit
grep 'client_max_body_size' /etc/nginx/nginx.conf   # Default 1m — too small for uploads

# 19. Header size limit
grep 'large_client_header_buffers' /etc/nginx/nginx.conf

# 20. Access log with upstream info
grep 'proxy_upstream_name\|proxy_upstream_addr' /etc/nginx/nginx.conf

5 Real-World Configs

Reverse Proxy for a Node.js App on 127.0.0.1:3000

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;

    access_log /var/log/nginx/api.example.com.access.log vhost;
    error_log  /var/log/nginx/api.example.com.error.log warn;

    client_max_body_size 20m;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-Id $request_id;
        proxy_set_header Connection "";

        proxy_connect_timeout 5s;
        proxy_send_timeout    60s;
        proxy_read_timeout    60s;
    }
}

Reverse Proxy for a Python/Flask Dev Server

upstream flask {
    server 127.0.0.1:5000;
    keepalive 16;
}

server {
    listen 443 ssl;
    server_name dev.example.com;

    location / {
        proxy_pass http://flask;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Connection "";
        proxy_read_timeout 120s;   # Dev servers tend to be slow
    }
}

Reverse Proxy with gRPC Passthrough

location /api/v1 {
    grpc_pass grpc://backend:50051;
    proxy_http_version 1.1;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Rate-Limited Reverse Proxy

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    listen 443 ssl;
    server_name api.example.com;

    location / {
        limit_req zone=api burst=20 nodelay;
        limit_req_status 429;
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }
}

WebSocket-Enabled Reverse Proxy

location /ws {
    proxy_pass http://backend/ws;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 3600s;      # Keep WebSocket alive for an hour
    proxy_send_timeout 3600s;
}

502 Bad Gateway — Decision Tree

502 → Check Nginx error.log
     ↓
     ├─ "connect() failed (111: Connection refused)"
     │  → Upstream not listening. Fix: restart upstream.
     │
     ├─ "connect() failed (110: Operation timed out)"
     │  → Firewall between Nginx and upstream, or network partition.
     │  Fix: iptables / security group.
     │
     ├─ "upstream prematurely closed connection"
     │  → Keepalive mismatch, or upstream crashed on request.
     │  Fix: `proxy_set_header Connection "";` and check upstream logs.
     │
     ├─ "upstream sent invalid header"
     │  → Protocol mismatch (HTTP vs HTTPS, HTTP/1 vs gRPC).
     │  Fix: align `proxy_pass` protocol with upstream.
     │
     └─ "no live upstreams"
        → All upstream servers failed. Check upstream health.

504 Gateway Timeout — Decision Tree

504 → Check upstream took how long?
     ↓
     ├─ < proxy_read_timeout → 502, not 504 (see above)
     │
     └─ > proxy_read_timeout (default 60s)
        → Slow upstream.
           Fix (a): optimize upstream (usually right answer)
           Fix (b): raise `proxy_read_timeout` deliberately
                    (with a comment explaining why)
           Fix (c): move long jobs to async (queue + poll)

502 vs 504 vs 499 — Quick Reference

Code	Meaning	Where to fix
502 Bad Gateway	Nginx couldn't reach or talk to upstream	Nginx config, upstream service
504 Gateway Timeout	Upstream too slow for `proxy_read_timeout`	Upstream performance, or bump timeout
499 Client Closed Request	User gave up before Nginx got a response	Nginx is fine — user/network issue
503 Service Unavailable	Upstream returned 503, or Nginx is overloaded	Upstream health, or `worker_processes`

Wrapping Up

502 and 504 are the two workhorses of reverse-proxy debugging. Half of them are "upstream not running" and "wrong port in proxy_pass" — the cheapest bugs to fix. The harder ones (keepalive, TLS mismatch, gRPC) all boil down to one question: what does Nginx think it's talking to, and what is the upstream actually talking back? Read the error.log string, match it against this decision tree, and you'll usually know what to change within a minute.

For related reading, see: - TCP Troubleshooting: TIME_WAIT / RST / SYN Half-Open — the layer below - Wireshark Basics: 30 Filters + Field Cheat Sheet — when logs aren't enough - CDN Origin & Cache: Origin Fetch, Preheat, Invalidation — 502 sometimes lives on the CDN side, not your Nginx

Nginx Reverse Proxy 502 / 504 Troubleshooting

The 30-Second Diagnosis

5 Real Failure Scenarios

1. Upstream Not Running

2. Wrong Address Behind proxy_pass

3. TLS Mismatch Between Nginx and Upstream

4. Keepalive Stale Connection

5. Backend Protocol Mismatch (HTTP/1.1 vs HTTP/2)

20-Command Cheat Sheet

5 Real-World Configs

Reverse Proxy for a Node.js App on 127.0.0.1:3000

Reverse Proxy for a Python/Flask Dev Server

Reverse Proxy with gRPC Passthrough

Rate-Limited Reverse Proxy

WebSocket-Enabled Reverse Proxy

502 Bad Gateway — Decision Tree

504 Gateway Timeout — Decision Tree

502 vs 504 vs 499 — Quick Reference

Wrapping Up

2. Wrong Address Behind `proxy_pass`