Debugging DNS Inside Kubernetes When nslookup Lies

The short answer

nslookup and dig use their own resolver logic. Your application uses glibc or musl. They read the same /etc/resolv.conf and apply it differently, particularly the ndots and search directives.

So nslookup postgres succeeding tells you almost nothing about whether your app can resolve postgres.

Test the way your app resolves:

getent hosts postgres                    # uses the same NSS path as glibc
python3 -c "import socket; print(socket.getaddrinfo('postgres', 5432))"

Then read the actual config:

cat /etc/resolv.conf

Tested on Kubernetes 1.32 with CoreDNS.

What resolv.conf means in a pod

search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal
nameserver 10.96.0.10
options ndots:5

search is a list of suffixes appended to unqualified names.

ndots:5 means: if a name contains fewer than 5 dots, treat it as relative and try each search domain before trying the name as-is.

That second one is the source of most confusion.

api.payments.com has two dots. Fewer than five. So the resolver tries:

api.payments.com.default.svc.cluster.local   NXDOMAIN
api.payments.com.svc.cluster.local           NXDOMAIN
api.payments.com.cluster.local               NXDOMAIN
api.payments.com.ec2.internal                NXDOMAIN
api.payments.com                             answer

Five queries for one hostname. And glibc issues A and AAAA in parallel, so ten packets.

nslookup does not do this by default. It treats the name as absolute. Which is exactly why it succeeds while your application is making ten queries and possibly timing out on one of them.

The failure modes

Intermittent slow resolution

Symptom: p99 latency spikes with no corresponding change in dependencies, and the profiler shows nothing because the time is spent waiting rather than computing.

Cause: the search domain amplification above, multiplied by traffic, saturating CoreDNS. When a UDP response is dropped, glibc waits its full timeout, which defaults to 5 seconds, before retrying.

One lost packet equals a five second request.

kubectl -n kube-system top pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 | grep -i "timeout\|refused"

Resolution works for some names and not others

The ndots boundary. A name with five or more dots is tried as absolute first. Fewer, and the search list runs first.

So svc.namespace.svc.cluster.local behaves differently from svc.namespace. If one works and one does not, ndots is involved.

Works in one namespace and not another

The first search domain is namespace specific: default.svc.cluster.local. A pod in production resolving the bare name postgres gets postgres.production.svc.cluster.local, not postgres.default.svc.cluster.local.

Cross namespace references need at least two labels:

postgres.default                                   works from anywhere
postgres.default.svc.cluster.local                 fully qualified
postgres                                           only within the same namespace

Only IPv6 fails, or connections take exactly 5 seconds

glibc sends A and AAAA queries in parallel over the same socket. Some older CNI and conntrack combinations drop one of the two responses, and the resolver waits out its timeout.

The classic symptom is that everything takes almost exactly 5 seconds. If you see that number, this is a strong suspect.

Mitigation is single-request-reopen in resolv.conf options, or disabling AAAA lookups if you have no IPv6.

Diagnosing properly

Get a shell with tools in the pod's network namespace:

kubectl debug -it mypod --image=nicolaka/netshoot --target=app

That shares the network namespace, so you see exactly what the application sees.

Then:

cat /etc/resolv.conf

# resolve the way the app does
getent hosts postgres

# see every query the resolver actually makes
strace -f -e trace=network getent hosts api.payments.com 2>&1 | grep -i sendto

# watch DNS traffic
tcpdump -i any -n port 53 -c 30

# query the cluster DNS directly, bypassing search logic
dig @10.96.0.10 postgres.default.svc.cluster.local +short

The strace line is the one that settles arguments. It shows the actual sequence of queries including every search domain attempt, and it makes the amplification visible. Same technique as using strace to find where time goes generally.

To compare tools directly:

dig +search postgres          # honours the search list
dig postgres                  # does not

If dig +search works and plain dig does not, you are relying on the search path, which is fine within a namespace and fragile across one.

The fixes

Lower ndots for services calling external hosts

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

Now api.payments.com is tried as absolute first. One query instead of five. Cluster internal short names still resolve through the search path.

This is the single highest value change for a service that mostly calls external APIs, and it typically cuts DNS query volume by 70 to 80 percent.

Fully qualify external hostnames

A trailing dot makes a name absolute regardless of ndots:

https://api.payments.com./v2/charge

Ugly and better supported than you would expect. Worth doing for your highest volume external endpoints even if you also set ndots.

NodeLocal DNSCache

A caching resolver as a DaemonSet on every node. Pods query a local address instead of crossing the network to CoreDNS, and it upgrades the upstream to TCP, which eliminates the silent-UDP-drop-plus-timeout failure entirely.

This is the most valuable infrastructure level fix and it is the one I would do first on any cluster with DNS problems.

Shorten the resolver timeout

dnsConfig:
  options:
    - name: timeout
      value: "1"
    - name: attempts
      value: "3"

Three fast attempts beat one five second stall. It converts a catastrophic tail into a mild one, and it is a backstop rather than a fix.

Scale and monitor CoreDNS

kubectl -n kube-system get deploy coredns

Two replicas is the default and it is frequently not enough. Watch request duration, error rate, and cache hit ratio. CoreDNS is infrastructure every single pod depends on and it is routinely unmonitored, which is the shared infrastructure blind spot in a nutshell.

Alpine and musl

Worth a separate note because the behaviour differs.

musl's resolver historically did not implement the search domain logic the same way, did not support more than a few nameservers, and handled the ndots option differently. Several versions had no search support at all in some paths.

The practical consequence: a service that resolves fine on a Debian base can fail on Alpine with the same resolv.conf. If you moved to Alpine for image size and DNS started behaving strangely, that is a real and known difference.

Options are installing musl-locales and a compatibility layer, using fully qualified names everywhere so search is irrelevant, or moving to a distroless or slim Debian base.

A checklist

• Test with getent hosts, not nslookup
• Read /etc/resolv.conf in the actual pod
• Count the queries with strace before assuming DNS is fine
• Check whether the name crosses a namespace boundary
• Look for exactly-5-second latencies, which point at resolver timeouts
• Check CoreDNS CPU and error rate, not just whether it is running
• Confirm whether your base image is glibc or musl