DevOps

The Thirty Seconds Between SIGTERM and SIGKILL

Rohit Bhadani

Rohit Bhadani

— 4 min read
Share
The Thirty Seconds Between SIGTERM and SIGKILL. Abstract devops illustration in orange and dark grey on debugly.dev

Every deploy produced a handful of 502s. Not many, never enough to trip an alert, and always dismissed as "the load balancer catching up". It was not the load balancer. It was our containers being killed mid request because nobody had written a signal handler.

The window is small and it is well defined. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (thirty by default), then sends SIGKILL. SIGKILL cannot be caught, blocked or handled. Whatever is in flight at that moment simply stops.

This was Kubernetes 1.32, Docker 27.5, Node 22.14 and Go 1.23. The sequence is identical everywhere.

What actually happens on a pod deletion

The order of operations is the part everyone gets wrong, because two things happen in parallel rather than in sequence.

  1. The API server marks the pod as terminating.
  2. In parallel: the endpoint controller removes the pod from the Service endpoints, and the kubelet sends SIGTERM to PID 1.
  3. Your application receives SIGTERM and starts shutting down.
  4. After the grace period, the kubelet sends SIGKILL to whatever is left.

Step 2 is the trap. Endpoint removal and SIGTERM are concurrent. Propagating the endpoint change to every kube proxy and every ingress controller takes time, often a second or two. During that window, traffic is still being routed to a pod that has already been told to die.

If your application stops accepting connections the instant SIGTERM arrives, you will drop requests that were already in flight through the network. This is not hypothetical and it is the cause of most "small number of errors during deploys" complaints.

The three defects I keep finding

Nothing handles SIGTERM

The default behaviour of SIGTERM is to terminate the process immediately. If you have not registered a handler, there is no graceful shutdown, there is only a slightly slower SIGKILL.

In Node, this is the entire fix:

const server = app.listen(3000);

process.on("SIGTERM", async () => {
  logger.info("SIGTERM received, draining");
  server.close(async () => {
    await db.end();
    process.exit(0);
  });
  setTimeout(() => process.exit(1), 25_000).unref();
});

server.close() stops accepting new connections and waits for existing ones to finish. The timeout is the escape hatch so a hung connection cannot eat the whole grace period.

In Go, http.Server.Shutdown(ctx) does the same job and the context is how you bound it.

PID 1 is a shell

This one is subtle and it is the reason docker stop takes ten seconds and then kills things.

CMD npm start

Docker runs that through /bin/sh -c, so PID 1 is the shell and your application is a child. Shells do not forward signals to their children by default. Your application never sees SIGTERM. Docker waits ten seconds, then sends SIGKILL to the process group.

Three fixes, in order of preference:

CMD ["node", "server.js"]

ENTRYPOINT ["tini", "--"]
CMD ["node", "server.js"]

CMD ["sh", "-c", "exec node server.js"]

The exec form in the third example replaces the shell process, so node becomes PID 1. If your Dockerfile has a shell form CMD, check this today. It is a one line fix for a defect that is invisible in every other respect.

The application drains but the platform does not wait

Even with a correct handler, if terminationGracePeriodSeconds is thirty and your longest request takes forty five, you will be killed at thirty. Either lengthen the grace period or bound your request duration. Both are legitimate. What is not legitimate is having a sixty second upstream timeout and a thirty second grace period, which guarantees that slow requests die during deploys. This is the same class of mismatch as every timeout should be shorter than the one above it.

The shutdown sequence I use

SIGTERM
  -> stop accepting new connections
  -> sleep 5 seconds   (let endpoints propagate)
  -> wait for in-flight requests, bounded at 20 seconds
  -> close database pools, flush queues, flush logs
  -> exit 0

That five second sleep looks like cargo cult and it is not. It covers the endpoint propagation window from step 2. Without it you will see a small, consistent trickle of errors at the very start of every rollout, which is exactly the pattern that gets dismissed as normal.

In Kubernetes, the same thing is better expressed as a preStop hook, because the hook runs before SIGTERM and therefore before your application starts shutting down:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]

Note that the preStop sleep is added to the grace period, not subtracted from it, so budget for it.

How to verify it works

Do not trust that it works. Send a signal and watch.

# locally, in one terminal
node server.js &
while true; do curl -s -o /dev/null -w "%{http_code} " localhost:3000/health; sleep 0.1; done

# in another
kill -TERM %1

If the curl loop shows anything other than 200s and then a clean stop, your drain is broken. In Kubernetes, the equivalent is a load generator running against the service while you kubectl rollout restart the deployment. Any non 2xx during the rollout is a defect, and it is a defect you can fix in an afternoon.

The same reasoning applies to anything that is not a web server. A queue worker needs to stop polling, finish the message it holds, and acknowledge it. A cron job needs to know whether it is safe to be interrupted mid run. The signal handler is where that logic goes, and if it is not there, the platform will make the decision for you.

Rollouts should be boring. If yours produce errors, the errors are telling you something specific about the thirty second window, and it is almost always one of the three defects above. For the adjacent problem of a process that will not start at all, see systemd service fails to start.

Read more