Event Loop Lag Is the Metric Nobody Alerts On

The alert said requests were timing out, and the dashboard said the service was fine. CPU sat at forty percent, memory was flat, no errors logged, no crash. The team restarted, it recovered, and the postmortem had no root cause, because none of the metrics they watched can see the failure. The process was not busy. It was blocked, and the difference between those two is the whole incident, and it is visible in exactly one metric: event loop lag.

I have lost on-call weeks to this exact shape of slow burn, so the order below is the order I fix it in.

Event loop lag is the delay between when a timer or IO callback was ready to run and when the loop got around to running it. It is the queueing delay of the single thread, and it is the metric that distinguishes a process that is working hard from a process that is stuck, which is the distinction every other metric misses.

This was Node 22.14, and the lag was measured with the built in performance hooks, which make it a five line addition and therefore inexcusable to lack.

Why the other metrics are blind

CPU utilisation measures how much of the time the core is executing, and a blocked loop can show low CPU, because the block is often a synchronous operation that is itself cheap per call but runs constantly, or a single long synchronous task that occupies the thread briefly per event but on every event, or a native call that does not count as JavaScript CPU at all. Forty percent CPU with a two hundred millisecond lag is a loop that spends its executed time fine and its waiting time stuck, and the utilisation averages over both and reports calm.

Memory measures the heap, and the classic block, a synchronous JSON parse of a large payload, allocates and frees within one tick, leaving no heap trace. The garbage collector variant shows up as a pause, which is also invisible to the heap size graph, because the size after the collection is normal.

Error rate measures failures the application chose to report, and a blocked loop does not report. It simply answers late, and the lateness is charged to the client as a timeout, which the client's dashboard attributes to the network or to your SLA, anywhere but the loop.

So the incident's signature is a service that is slow while every resource graph is calm, and the only graph that moves is the one nobody has.

Measuring it

The runtime exposes the loop's own delay directly:

const { monitorEventLoopDelay } = require("node:perf_hooks");
const h = monitorEventLoopDelay({ resolution: 20 });
h.enable();
setInterval(() => {
  metrics.set("event_loop_lag_p99_ms", h.percentile(99) / 1e6);
  metrics.set("event_loop_lag_max_ms", h.max / 1e6);
  h.reset();
}, 10000);

The histogram's high percentiles and its max are the numbers that matter, because lag is a tail phenomenon: the median is always near zero, and the p99 and the max are the blocked ticks. Alerting on the mean reproduces the blindness, which is the same wrong statistic as p99 and why averages lie, one thread deep.

The measurement costs almost nothing, because it is a timer comparing scheduled versus actual fire time, which is the definition of the quantity, and it runs in every process, in production, permanently.

Reading it in the incident

When the timeout alert fires, the loop lag graph partitions the incident in one glance.

Lag high with CPU high: the loop is saturated by work, a hot synchronous path or too much per tick work, and the fix is moving the work off the thread, the serialisation story in the endpoint that spent nine hundred milliseconds building JSON.

Lag high with CPU low: the loop is blocked by something that does not burn JavaScript CPU, a synchronous native call, a large synchronous IO, a DNS or fs call made synchronously, or a garbage collection pause. The fix is finding the blocking call, and the lag graph's periodicity often names it, a spike every N seconds is a cron or a GC cadence.

Lag normal: the loop is fine and the slowness is downstream, the database, the network, the dependency, and the investigation leaves the process entirely.

That three way partition is the diagnosis, and it arrives from one metric that most services do not export.

The common causes, in practice

A few causes produce most high lag incidents, and they are worth recognising.

The synchronous parse or stringify of a large payload, per request, on the hot path, which is the serialisation tax again, and which shows as lag proportional to payload size.

The regex on unbounded input that backtracks pathologically, a single request's pattern blocking the thread for seconds, which is the incident that restarts fix and that returns with the next malicious input.

The large synchronous filesystem or crypto call made "because it is only at startup", which is fine until the thing restarts under load and the startup runs while traffic arrives.

The garbage collection pause from a heap that is large and churny, where the collector stops the loop, and the fix is the heap hygiene from the memory leak work in Node memory leaks, same mechanism, different costume.

The takeaway

A Node process has one thread for all of its callbacks, and that thread's queueing delay is the service's responsiveness, and the delay is measured by event loop lag, which no resource metric can see. Export the p99 and max permanently, alert on them, and use the lag versus CPU partition as the first branch of every latency incident.

CPU says how hard the loop works. Lag says how long everything waits. The incident where the two disagree is the incident where the loop is blocked, and the dashboard without lag is a dashboard that cannot see the only thread there is.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud