The Outage Caused by a Successful Health Check

The database had a 40 second blip. A failover, handled automatically, the kind of thing that should produce a brief spike in errors and then recovery.

Instead it produced a 14 minute outage, and the database was healthy for 13 of those minutes.

What happened

The liveness probe looked like this:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

And /health did this:

app.get("/health", async (req, res) => {
  try {
    await db.query("SELECT 1");
    await redis.ping();
    const upstream = await fetch(`${PAYMENTS_URL}/health`, { signal: AbortSignal.timeout(2000) });
    if (!upstream.ok) throw new Error("payments unhealthy");
    res.json({ status: "ok" });
  } catch (err) {
    res.status(503).json({ status: "unhealthy", error: err.message });
  }
});

That looks thorough. It is checking the things the service depends on. It reads like good practice.

During the 40 second database blip, every replica's health check failed. Three consecutive failures at 10 second intervals is 30 seconds, which fits inside 40 comfortably.

So Kubernetes killed all twelve replicas. Simultaneously.

Then they all started at once, all tried to connect to a database that was still recovering, all exhausted the connection pool at the same moment, all failed their startup, and got killed again. The restart loop took longer to converge than the original incident by a factor of twenty.

The mistake

A liveness probe should answer one question: is this process wedged and does it need to be killed?

That is all. Not "is everything this service depends on working". Not "can this service currently do useful work". Just: is this process irrecoverably stuck.

Because the only action Kubernetes takes on a liveness failure is to kill the container. So the question you are really answering is "would restarting this process help?"

If the database is down, restarting your process does not help. It makes things worse, because a cold process has no warm connection pool and no cache, and twelve of them starting simultaneously is a thundering herd against something already struggling.

Liveness versus readiness

The distinction that matters, and the one this outage was about.

Liveness failure kills the pod. Use it for: deadlock, an event loop that has stopped, a process that has entered an unrecoverable state. Things a restart genuinely fixes.

Readiness failure removes the pod from the load balancer. The process keeps running. Use it for: dependencies unavailable, cache still warming, currently overloaded. Things that resolve on their own.

Readiness is the right place for a dependency check, because the action it triggers is proportionate. Stop sending traffic, keep the process alive, resume when healthy.

Rewritten:

// liveness: is this process alive
app.get("/livez", (req, res) => {
  res.json({ status: "ok" });
});

// readiness: can this process serve traffic right now
app.get("/readyz", async (req, res) => {
  const checks = await Promise.allSettled([
    db.query("SELECT 1"),
    redis.ping(),
  ]);

  const failed = checks.filter(c => c.status === "rejected");
  if (failed.length) {
    return res.status(503).json({
      status: "not ready",
      failed: failed.map(f => String(f.reason)),
    });
  }
  res.json({ status: "ready" });
});

The liveness endpoint returns 200 unconditionally. That looks lazy and it is correct for most services: if the HTTP server can answer, the process is not wedged.

If your service can genuinely deadlock, check for that specifically rather than checking dependencies:

app.get("/livez", (req, res) => {
  const lag = eventLoopLagMs();
  if (lag > 5000) return res.status(503).json({ status: "event loop blocked", lag });
  res.json({ status: "ok" });
});

That is a real liveness check. It detects a condition a restart actually fixes.

The upstream check was worse

Note the original also checked a downstream service's health endpoint.

If the payments service is unhealthy, every service that checks it becomes unhealthy, and everything checking those becomes unhealthy. One service's problem propagates through the health check graph as though it were a dependency failure, and the whole system restarts together.

Never check a transitive dependency in a health check. Your health is about you. If a dependency is down, you handle that with timeouts, circuit breakers, and degraded responses, not by declaring yourself dead.

The startup problem, which is separate

Once liveness stopped checking dependencies, a second issue appeared: the service takes about 40 seconds to warm up, and the liveness probe was starting to check at 10 seconds.

The old fix is a large initialDelaySeconds, which is a bad trade because it also delays detection of a genuinely wedged process for the whole startup window.

The right tool is a startup probe:

startupProbe:
  httpGet: { path: /livez, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5          # allows up to 150s to start

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

The startup probe suspends liveness until the process comes up once. After that, liveness runs at its normal aggressive cadence. You get patience at startup and fast detection afterwards, which the initialDelaySeconds approach cannot give you.

This is one of the more common causes of pods stuck in CrashLoopBackOff and it is entirely a configuration problem.

What else changed

A PodDisruptionBudget, so not everything can be unavailable simultaneously:

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 50%
  selector:
    matchLabels: { app: checkout-api }

This does not prevent liveness kills, which is worth knowing, and it does protect against voluntary disruptions like node drains.

Connection retry with backoff and jitter at startup. Twelve replicas starting simultaneously and all retrying on the same interval is a synchronised herd. Jitter spreads them.

A degraded mode. The service now serves cached data with a header indicating staleness when the database is unavailable, rather than failing entirely. For a read heavy endpoint this converts an outage into a degradation, and it was the highest value change of the lot.

The general lesson

The health check was written by somebody trying to be thorough. Every individual check in it was reasonable. The problem was that the checks were wired to an action that did not match them.

The question to ask of any automated remediation is: what does this do, and is that the right response to what I am detecting?

Killing a process is the right response to a wedged process. It is the wrong response to a dependency outage. Same detection, same signal, and the correct action depends entirely on the cause, which is why lumping them into one endpoint produced an outage.

The related failure I now look for everywhere: automation that responds to a symptom rather than a cause, and does so at scale, simultaneously. Autoscalers that scale up in response to latency caused by a downstream limit. Retries that amplify load on a struggling service. Circuit breakers that all open and close in unison.

All of them are the same shape as this one, and all of them turn a small problem into a large one faster than a human would.