The WebSocket That Died Every Sixty Seconds and Nobody Owned Up

A live order feed, WebSocket, working perfectly on every developer machine and dropping in production. The reconnect logic covered it well enough that nobody filed a ticket for two weeks, until someone noticed the reconnect count in the dashboard was 1,400 per hour.

The connections were not failing randomly. They were failing at sixty seconds, almost exactly, over and over.

The symptom, stated precisely

socket.addEventListener('close', (e) => {
  console.log(new Date().toISOString(), 'closed', e.code, e.reason, 'clean:', e.wasClean);
});

2026-08-30T09:14:03.118Z closed 1006  clean: false
2026-08-30T09:15:03.402Z closed 1006  clean: false
2026-08-30T09:16:03.771Z closed 1006  clean: false

Code 1006 means abnormal closure with no close frame received. Neither side sent a proper close. Something in the middle severed the connection and told nobody.

The sixty second regularity was the entire diagnosis sitting in plain sight, and I spent a day on other theories first.

Tested on nginx 1.27, Node 22.14, Chrome 133.

The hypotheses that were wrong

Hypothesis one: the application is throwing

Reasonable first guess. An unhandled error in the message handler could tear down the connection.

Ruled out quickly. Server logs showed no errors, and the server side close handler fired after the client's, meaning the server learned about it second. It was not the origin.

Hypothesis two: a load balancer draining connections during deploys

Plausible, because we deploy often. But deploys happen a few times a day and this was happening every sixty seconds continuously, including overnight with no deploys at all.

I checked anyway, because it was cheap. No correlation.

Hypothesis three: client network instability

Corporate wifi, VPNs, laptops sleeping. This is the theory everyone reaches for and it is almost always wrong when the interval is regular.

The thing that killed it: I reproduced the disconnect from a bare EC2 instance on a wired connection with a script, and it dropped at sixty seconds too. Networks are not that punctual.

That was the moment the interval stopped being background detail and became the clue.

The breakthrough

A precise, repeating interval means a timeout, and a timeout means a configured value. Something between client and server had a sixty second setting.

I mapped the path:

browser -> CDN -> nginx ingress -> node service

Then read the timeout defaults for each hop. nginx's proxy_read_timeout defaults to sixty seconds.

That setting means: if the upstream sends nothing for sixty seconds, close the connection. For a normal HTTP request that is a sensible safety net. For a WebSocket, which is idle by design between events, it is a guillotine on a timer.

Our feed was busy during trading hours and nearly silent outside them. The disconnects clustered exactly when the feed was quiet, which also explained why nobody reproduced it locally: on a developer machine there is no proxy, and the browser talks to the server directly.

# the confirmation, from a shell
timeout 90 websocat wss://api.example.com/feed -v
# closed at ~60s with no data flowing

Idle for sixty seconds, connection closed. Send anything at all, and the timer resets.

What I changed

Raised the proxy timeout and made the upgrade explicit.

location /feed {
  proxy_pass http://app_upstream;
  proxy_http_version 1.1;

  # required, or the upgrade never happens at all
  proxy_set_header Upgrade    $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_set_header Host       $host;

  # a WebSocket is idle by design, so the read timeout must exceed the ping interval
  proxy_read_timeout  3600s;
  proxy_send_timeout  3600s;

  proxy_buffering off;
}

proxy_http_version 1.1 is mandatory. nginx defaults to HTTP/1.0 upstream, which has no upgrade mechanism, so without it the handshake fails outright rather than dying later.

proxy_buffering off matters for streaming. With buffering on, nginx accumulates output and delivers it in chunks, which makes a live feed arrive in bursts and look like lag.

Added application level heartbeats, which is the real fix.

Raising a timeout is fragile because it depends on every hop agreeing, and you do not control a corporate proxy or a mobile carrier's NAT. A ping keeps the connection non idle everywhere at once.

const HEARTBEAT_MS = 25000;

wss.on('connection', (ws) => {
  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });
});

setInterval(() => {
  for (const ws of wss.clients) {
    if (!ws.isAlive) { ws.terminate(); continue; }
    ws.isAlive = false;
    ws.ping();
  }
}, HEARTBEAT_MS);

Twenty five seconds is deliberate. Many intermediaries use thirty or sixty; staying under thirty clears nearly all of them. This also gives the server a genuine liveness check rather than assuming an open socket is a working one.

Fixed the reconnect to back off.

The original reconnected immediately, so 1,400 drops per hour became 1,400 handshakes per hour, each one a TLS negotiation. Exponential backoff with jitter, capped at thirty seconds:

const delay = Math.min(30000, 1000 * 2 ** attempt) * (0.5 + Math.random() * 0.5);

Jitter matters. Without it, every client that dropped together reconnects together, and you build a thundering herd on top of an outage.

After the change: reconnects fell from 1,400 per hour to under 20, and those were genuine client network events.

What I would do differently

Treat a regular interval as a configuration value immediately. Sixty seconds, thirty seconds, five minutes, 350 seconds. These are defaults somebody chose. Networks fail irregularly; configuration fails punctually. I had the number in the first log I read.

Enumerate the hops before theorising. I spent a day on the two endpoints without writing down what was between them. The list took four minutes once I bothered, and the answer was in it.

Distrust "works locally" as evidence about the application. It is usually evidence about the path, which is the same lesson as a Docker build that works locally and fails in CI: the code is identical and the environment is not.

Look at the close code. 1006 specifically means no close frame arrived, which rules out both endpoints doing an orderly shutdown. A 1001 would have meant the server going away, 1011 a server error. The code narrowed it to a middlebox on the first line of the first log, and I did not read it properly.

If you want the general version of this, I wrote about setting timeouts so they nest correctly, which is the same problem seen from the design end rather than the debugging end.