The Ingress Annotation That Silently Changed Your Timeout

The service was tested against its own nginx configuration, where the timeouts were set deliberately, and it behaved. In the cluster, behind the ingress controller, the same service produced 413s on uploads that passed locally and 504s on reports that finished fine in staging. The difference was a second nginx, the one the controller runs, with its own defaults, and the defaults were the configuration, because nobody had written an annotation to override them.

I have run this failure in production myself, and the fix below is the one I now reach for first.

An ingress controller is a proxy you do not see, running a configuration you did not write, whose defaults apply to every service that does not explicitly override them. This post is the list of the defaults that bite, and the discipline for owning them.

This was nginx ingress on Kubernetes 1.32, controller defaults as shipped, and the app's own nginx 1.27 as the tested baseline.

The body size that rejects your upload

The controller's default maximum request body is small, commonly one megabyte, while a standalone nginx defaults to the same number but your tested config had raised it. Uploads above the controller's limit are rejected with 413 at the ingress, before the application ever sees them, which is why the application logs showed nothing and the local tests passed.

This is the two limits problem from 413 request entity too large with two limits, with the ingress as the limit you forgot. The annotation exists to raise it per ingress, and the fix is setting it where the upload is expected, not globally, because a global raise removes a useful defence from every other route.

The timeouts that are not yours

The controller's proxy read, send and connect timeouts have their own defaults, typically sixty seconds, applied unless annotated. A report that runs ninety seconds succeeds against the tested nginx with its raised timeout and dies at the ingress with a 504 at sixty, while the application, unaware, finishes at ninety and writes to a closed connection, the orphaned work from 504 gateway timeout while your application was still working.

The ladder lesson applies with an extra rung now: the client's timeout, the ingress's timeout and the app's timeout must agree in shrinking order, and the ingress is the rung most teams never set, so it defaults to a number chosen by the controller's authors, not by anyone who knows the workload.

Buffering that changes the failure mode

Proxy buffering is on by default, meaning the ingress reads the whole response from the app before or while sending it to the client. For large or streaming responses this changes both memory use and failure shape: a slow client makes the ingress hold the buffered body, and a response larger than the buffer spills to disk with its own limits. Streaming endpoints, server sent events and long polling, misbehave behind buffering unless the annotation disables it for those routes, because buffering turns a stream into a file the client waits for.

The rewrite and path surprises

Annotations that rewrite the target path are convenient and dangerous, because the rewrite happens at the ingress and the application sees the rewritten path, so logs and redirects disagree with what the client sent. A trailing slash difference in a rewrite turns a working route into a redirect loop or a 404 that reproduces nowhere except through the ingress, the same composition bug class as the path parameter that failed on a comma, where the middle layer's interpretation is the defect.

The protocol assumption

The controller's view of whether the connection to the backend is HTTP or HTTPS, and which port it uses, comes from annotations and service ports. A backend serving HTTPS on a port the ingress assumes is HTTP produces handshake garbage and 502s that look like application crashes, the protocol disagreement from SSL wrong version number, relocated to the ingress edge.

The discipline for owning the invisible nginx

The defaults are fine for the average service, which is why they exist, and the defect is not the defaults but the absence of an inventory of where they differ from what the service needs.

Make the effective configuration visible. Render or dump the controller's generated nginx configuration for the routes that matter, and read the timeouts, body size and buffering as they actually are, not as the application assumes. The generated config is the ground truth, and it is the diff against the app's tested config that lists the surprises.

Set the annotations per route, in the same repository as the service, reviewed with the service, so the proxy contract travels with the code it proxies. An annotation that lives in a separate manifest, changed by a different team, is a configuration fork, and forks disagree.

Alert on the ingress's 413 and 504 rates per route, because both are the controller enforcing a default the service did not choose, and the alert is the inventory's tripwire when a new default or a new route reintroduces a mismatch. A controller upgrade that changes a shipped default is exactly the event this catches, and controller upgrades are the one time the invisible nginx is visibly rewritten, which is when the diff against your tested configuration matters most.

The takeaway

An ingress controller is a second proxy with its own defaults for body size, timeouts, buffering, rewrites and protocol, and those defaults are the live configuration for every route that does not override them. Test against the generated configuration, set the overrides per route in the service's repository, and alert on the controller's rejection codes, because the nginx you tested is not the nginx that serves.

The ladder of agreeing timeouts now has a rung you do not own by default, and owning it starts with admitting it exists. The client side view of the 504 it produces is the curl timing template in the curl flag that tells you which part was slow.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud