How the Libuv Threadpool Starves Your DNS Lookups

The DNS lookups slowed to seconds while the event loop lag stayed flat and the network was clean, which is the combination that sends the investigation to the resolver and the NIC, both innocent, because the stall was not in the network, it was in the queue, a queue inside the process, four threads deep, shared by every piece of work Node delegates off the event loop, and the lookup was waiting behind a slow file read that had nothing to do with DNS.

I have lost on-call weeks to this exact shape of slow burn, so the order below is the order I fix it in.

This is the mechanics of the libuv threadpool, the small fixed pool that serves the process's delegated blocking work, and of the starvation that turns one slow filesystem call into a DNS outage, because the two share a queue neither knows about.

This was Node 22.14 with the default pool size, and the incident is the pool's arithmetic, which is small, fixed, and shared, which are the three properties that make it a bottleneck.

The pool, and what it serves

Node's event loop must not block, so the operations that would block, filesystem calls, DNS resolution, some crypto, some compression, are handed to a threadpool of native threads, four by default, which execute them and return the result to the loop. The pool is therefore the waiting room for the process's blocking work, and its size, four, is a default chosen for a general workload, and its sharing is total, a file read and a DNS lookup are the same kind of customer, and they queue together.

The stall is queueing theory at four servers: when the arrival of blocking work exceeds the pool's service rate, the queue grows, and every customer, including the DNS lookup that arrived after a slow file read, waits the sum of the work ahead of it, and the wait is added to the lookup's latency, invisibly, because the lookup itself, once it reaches a thread, is fast.

Why the event loop looks fine

The event loop lag measures the loop's own queue, the timers and callbacks on the main thread, and the threadpool's queue is off the loop, so the loop lag stays flat while the pool queue grows, which is the confusing signature: the process's main metric is healthy and the delegated work is stalled, and the two queues are separate, so the healthy one says nothing about the starving one. This is the lag metric's blind spot from event loop lag is the metric nobody alerts on, and the shared property is the one worth fixing.

The DNS lookup, specifically, is a pool customer on most platforms for the getaddrinfo call, so its latency includes the pool wait, and the lookup that "takes two seconds" took two milliseconds of resolution and 1998 of queue, and the network, blamed first, did nothing wrong, which is the partition the latency spike that was DNS post draws at the resolver, while this one draws it inside the process, one hop earlier.

The trigger: one slow customer

The starvation needs a slow customer to seed the queue, and the filesystem provides it: a read of a large file, a log tail, a config parse of a big blob, each occupies a thread for its duration, and with four threads, a handful of concurrent slow reads saturates the pool, and every subsequent lookup, and every other delegated call, queues behind them. The trigger is therefore a workload change, a new feature that reads files, which starves the DNS of an unrelated feature, which is the coupling, two features sharing four threads without knowing it.

This is the noisy neighbour at the process level, per your database got slower and nothing in your code changed, which is why the two fixes look alike.

Seeing the queue

The pool's queue is observable, which is the fix's first half: libuv exposes the pool's state, and the metrics of queued and active work, sampled, give the queue length over time, and the lookup latency decomposed into queue wait and execution is the diagnosis, because the execution is flat while the wait spikes, which is the starvation's signature, and the spike aligns with the file reads, which names the slow customer.

The decomposition is the instrument: a DNS latency metric alone says "slow", the decomposed metric says "queued", and the queue is the pool, and the pool's customers are the file reads, and the chain is the diagnosis, in three hops, each named.

The fix, in order

Size the pool to the workload. The pool size is an environment variable, and raising it from four adds servers to the queue, which shrinks the wait for the same arrival, and the right size is the observed concurrent blocking work, which the queue metric provides, so the size is a measured decision, not a default, and the default is the bug.

Move the slow customer out of the pool. The large file read does not belong in the pool that serves DNS; it can run in a worker process, or stream instead of reading whole, or use the async filesystem where available, so the slow customer leaves the shared queue, which is the isolation fix, and isolation, not sizing, is the durable fix, because sizing shares the pool forever while isolation removes the coupling.

Cache the lookups. The DNS result is cacheable with its TTL, and a resolver cache removes the repeated lookups from the pool entirely, so the pool's arrival rate drops, which is the demand side, and the cache, per the discipline elsewhere, is keyed on the name and honoured against the TTL, so the freshness is bounded.

Alert on the pool queue. The queue length, alerted at a small threshold, is the tripwire, because the starvation is visible in the queue before it is visible in the lookup latency, and the queue is the cause, and alerting on the cause is the difference between a page and a postmortem.

The rule

Node delegates its blocking work, file and DNS among it, to a fixed pool of four threads, and the pool is a shared queue, so one slow file read seeds a queue that every lookup then joins, stalling the DNS while the event loop, on its own queue, reports health. Size the pool from the measured concurrent work, move the slow customer out, cache the lookups, and alert on the queue.

The two second lookup was a two millisecond resolution wearing a two second wait, and the wait was the pool, and the pool was four threads shared by strangers in the same process, which is the noisiest neighbourhood there is, because the neighbours are your own features, and they never introduce themselves.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud