Readiness, Liveness and the Probe That Restarted a Healthy Pod

The service's p99 spiked, and the response was a restart storm: Kubernetes killed the pods, the pods came back, the spike returned, and the cycle repeated, each restart dumping in flight requests and resetting caches, which deepened the very latency the probes were reacting to. The postmortem's uncomfortable finding was that the pods were healthy. The work was slow, not broken, and the liveness probe, which shared the single event loop with the work, could not get CPU to answer in time, so the orchestrator concluded death and administered the one treatment it knows, which is the treatment that made the condition worse.

This is the mechanics of how Kubernetes decides a pod is unhealthy, and why the decision can be wrong in the most expensive direction: restarting a healthy, saturated process.

This was Kubernetes 1.32 with a Node 22.14 single threaded service, and the probe configuration was the default shaped one that most manifests carry.

The three judgements Kubernetes makes

Kubernetes does not measure health directly. It asks the process questions over HTTP or TCP, and the questions are of three kinds.

Startup. Has the process finished initialising. The startup probe gates the others, so a slow boot is not mistaken for a dead process.

Readiness. Are you able to serve right now. A failing readiness probe removes the pod from the service's endpoints, so traffic goes elsewhere, but the pod is not killed. Readiness is a traffic valve, not a verdict on life.

Liveness. Are you alive at all. A failing liveness probe restarts the container, because the assumption is a deadlocked or wedged process that only death can cure.

The asymmetry is the whole design: readiness sheds load, liveness kills. The incident happens when liveness fires on a process that is not wedged but merely busy, because killing a busy process discards its work and its caches, and the replacement inherits the same traffic with a cold start, which is a latency tax that can re trigger the probe, a feedback loop where the cure is the cause.

Why a healthy pod fails liveness

The liveness probe is answered by the same process it judges. In a single threaded runtime, the probe handler waits behind the same queue as the work, so when the event loop is saturated, the probe's latency is the work's latency, and a timeout configured for an idle process fires under load. The probe then reports a healthy process as dead, because the probe measures responsiveness, and responsiveness is exactly what saturation removes, while aliveness, the thing liveness is supposed to measure, is intact.

This is the distinction the configuration forgets: a saturated loop is busy, a wedged loop is stuck. The busy loop will recover if you shed load. The stuck loop will not recover at all. Liveness is the right tool only for the stuck case, and the probe as commonly configured cannot tell the two apart, because both answer slowly or not at all.

The same blindness appears in the health check that succeeds while the service fails, per outage caused by a successful health check, but inverted: there the probe was too shallow to see the failure, here the probe is too coupled to the work to see the health.

The restart feedback loop

Once liveness fires on a busy pod, the restart makes the fleet smaller, so the surviving pods take more traffic, saturate further, and fail their probes too, cascading the restarts. Each new pod also starts cold, with empty caches and unwarmed connections, so its first seconds are its slowest, precisely when the probe begins asking. The loop is a textbook positive feedback, and the configuration parameter that controls it is the liveness timeout, which was set for the idle case.

This is the canary lesson from a canary deployment that shipped a broken release anyway, at the pod level: the signal was too small and too coupled to represent the truth, and the automated response amplified the error.

The configuration that separates busy from stuck

The fix is to make liveness unable to fire on busyness, and to let readiness carry the load shedding.

Liveness must be cheap and independent. The liveness handler should answer from a dedicated, trivial path that does not queue behind the work, a separate thread or a precomputed flag, so a saturated loop still answers liveness quickly. If the probe can be answered without the event loop, it measures aliveness, which is its job. A liveness endpoint that does real work is a liveness endpoint that fires under load.

Liveness timeout and failure threshold must tolerate the worst legitimate stall. Set them from the observed p99 of the probe under load, with margin, so busyness never crosses them. The threshold is a statement about what stall duration implies wedged, and it should be long, because a restart is expensive and should be rare.

Readiness carries the valve. Let readiness fail fast under saturation, removing the pod from traffic so it can drain and recover, which is the non destructive response to busyness. Readiness shedding load is the system helping the pod recover. Liveness killing it is the system assuming recovery is impossible.

Startup probe for the cold window. Give the boot its own generous startup probe, so the cold start's slowness is not judged by liveness, which is the graceful shutdown lesson at the other end of the lifecycle.

The rule I keep

Kubernetes judges health by asking the process questions, and the liveness question, answered by the same loop as the work, measures responsiveness, not aliveness, so a saturated healthy pod fails it and gets the one treatment that deepens saturation. Make liveness cheap and independent of the work, tolerant of the worst legitimate stall, let readiness shed the load, and reserve the restart for the process that is truly wedged.

The restart storm was not Kubernetes malfunctioning. It was Kubernetes applying a correct rule to a wrong signal, and the signal was wrong because the probe shared the queue it was supposed to judge, which is the oldest measurement error there is, wearing a manifest.