504 Gateway Timeout While Your Application Was Still Working
The alert said 504s on a report endpoint. The application logs for the same requests showed success, at a timestamp after the 504 had already been returned. The proxy had declared the request dead while the application was, in fact, still working, and then finished a job whose client had already left.
A 504 is not an application failure. It is a proxy's statement that the upstream took longer than it was willing to wait. The application never gets to disagree. This mismatch, proxy timeout shorter than real work, is one of the most common and least understood sources of "the request failed but the work happened" bugs.
This was nginx 1.27 in front of a Node 22.14 service, with proxy_read_timeout at the default sixty seconds.
The short answer
proxy_read_timeout is the time nginx will wait between two successive read operations from the upstream. If your application thinks for longer than that without sending any bytes, nginx closes the client connection with a 504 and logs the upstream as timed out. The application, unaware, continues and completes, writing to a socket that is gone.
So the request "failed" at the edge and "succeeded" in the service. Any side effect the request performed actually happened. The client just never found out.
Why the default is a trap
Sixty seconds feels generous until you have real workloads. A report over a large table, a batch import, a cold analytics query, an ML inference on a big model, all routinely exceed a minute. The default was chosen for interactive requests, where a minute of silence means something is hung, not for the long tail of legitimate slow work.
The trap is that the endpoint works fine in development and staging, where the data is small, and fails in production, where it is real. The timeout is constant. The work is not.
The two directions of fix
There are exactly two honest directions, and picking between them is a design decision.
Make the proxy wait. Raise the timeout for the routes that do real work. This is fine for bounded work, and it is the simple fix. The danger is raising a global timeout so high that genuinely hung upstreams tie up proxy connections for a long time, converting a hung worker into a resource leak at the edge.
Make the work asynchronous. For unbounded work, the correct shape is to return quickly with a job identifier and let the client poll or receive a webhook. The proxy never waits for the work, because the work is no longer in the request path. This is the fix that scales, and it is the same lesson as moving long operations out of the request in reviewing timeouts and cancellation.
Between the two is a middle option that works for medium work: have the application send periodic heartbeat bytes so the proxy's read timeout never fires. This keeps the connection alive by never letting the silence exceed the timeout. It is a legitimate pattern for streaming responses, but it papers over the fundamental issue for very long work.
The timeout ladder
The deeper defect is usually a ladder of timeouts that do not agree. The client has a timeout, the proxy has a timeout, the load balancer has a timeout, and the application has a timeout, and they were each set by different people at different times.
The rule is that each layer's timeout should be longer than the one above it, with a margin, so that the layer closest to the user gives up first and the layers below never orphan work the user has already abandoned. When the proxy times out before the client, you produce exactly this 504 plus orphaned work. The ladder and its mismatches are covered in every timeout should be shorter than the one above it.
Distinguishing 504 from 502
Both are gateway errors and both are blamed on the application, but they differ usefully.
A 502 means the proxy could not get a valid response from the upstream at all, usually because the upstream crashed, refused the connection or sent garbage. The upstream is the suspect.
A 504 means the proxy connected and waited, and the upstream was silent too long. The upstream may be perfectly healthy and simply slow. This is the distinction in nginx 502 bad gateway, and conflating the two sends you to read crash logs for an application that did not crash.
Seeing it in the logs
The nginx error log for a 504 says upstream timed out and gives the time waited. The access log shows the 504 and the upstream response time field, which reads as the timeout value rather than a completion. Meanwhile the application access log shows a 200 with a duration longer than the proxy's timeout.
That pair of log lines, edge 504 at sixty seconds and app 200 at seventy four, is the entire diagnosis. Once you see them together, no further theory is needed.
What I changed
For the report endpoint I moved it to an async job with a poll endpoint, because reports are unbounded and belong out of the request path. For the bounded interactive endpoints I aligned the ladder: client forty five, proxy fifty, upstream work bounded to forty. And I added a metric for upstream response time at the proxy so that any endpoint creeping toward the proxy timeout pages before it crosses it.
The rule
A 504 is the proxy's timeout, not the application's verdict. When the edge says failure and the app says success, the work happened and the timeout was simply shorter than the work. Fix by making the work bounded and asynchronous, or by aligning the timeout ladder so the user side gives up first.
The orphaned work side of this defect, where the application keeps doing work nobody wants, is the streaming and disconnect story in EPIPE broken pipe.
