> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # Debugging DNS Inside Kubernetes When nslookup Lies - URL: https://debugly.dev/kubernetes-dns-debugging-nslookup/ - Published: 2025-12-23T09:10:00.000Z - Updated: 2026-10-17T05:59:03.000Z - Description: nslookup resolves and your application cannot connect. The two use different resolvers, and the difference is where the bug lives. - Author: Rohit Bhadani - Tags: Kubernetes, Networking, DevOps, Debugging ## The quick version `nslookup` and `dig` use their own resolver logic. Your application uses glibc or musl. They read the same `/etc/resolv.conf` and apply it **differently**, particularly the `ndots` and `search` directives. So `nslookup postgres` succeeding tells you almost nothing about whether your app can resolve `postgres`. Test the way your app resolves: ```bash getent hosts postgres # uses the same NSS path as glibc python3 -c "import socket; print(socket.getaddrinfo('postgres', 5432))" ``` Then read the actual config: ```bash cat /etc/resolv.conf ``` Tested on Kubernetes 1.32 with CoreDNS. ## What resolv.conf means in a pod ``` search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal nameserver 10.96.0.10 options ndots:5 ``` **`search`** is a list of suffixes appended to unqualified names. **`ndots:5`** means: if a name contains fewer than 5 dots, treat it as relative and try each search domain before trying the name as-is. That second one is the source of most confusion. `api.payments.com` has two dots. Fewer than five. So the resolver tries: ``` api.payments.com.default.svc.cluster.local NXDOMAIN api.payments.com.svc.cluster.local NXDOMAIN api.payments.com.cluster.local NXDOMAIN api.payments.com.ec2.internal NXDOMAIN api.payments.com answer ``` Five queries for one hostname. And glibc issues A and AAAA in parallel, so ten packets. `nslookup` does not do this by default. It treats the name as absolute. Which is exactly why it succeeds while your application is making ten queries and possibly timing out on one of them. ## The failure modes ### Intermittent slow resolution Symptom: p99 latency spikes with no corresponding change in dependencies, and the profiler shows nothing because [the time is spent waiting rather than computing](https://debugly.dev/latency-spike-that-was-dns/). Cause: the search domain amplification above, multiplied by traffic, saturating CoreDNS. When a UDP response is dropped, glibc waits its full timeout, which defaults to **5 seconds**, before retrying. One lost packet equals a five second request. ```bash kubectl -n kube-system top pods -l k8s-app=kube-dns kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 | grep -i "timeout\|refused" ``` ### Resolution works for some names and not others The `ndots` boundary. A name with five or more dots is tried as absolute first. Fewer, and the search list runs first. So `svc.namespace.svc.cluster.local` behaves differently from `svc.namespace`. If one works and one does not, `ndots` is involved. ### Works in one namespace and not another The first search domain is namespace specific: `default.svc.cluster.local`. A pod in `production` resolving the bare name `postgres` gets `postgres.production.svc.cluster.local`, not `postgres.default.svc.cluster.local`. Cross namespace references need at least two labels: ``` postgres.default works from anywhere postgres.default.svc.cluster.local fully qualified postgres only within the same namespace ``` ### Only IPv6 fails, or connections take exactly 5 seconds glibc sends A and AAAA queries in parallel over the same socket. Some older CNI and conntrack combinations drop one of the two responses, and the resolver waits out its timeout. The classic symptom is that everything takes almost exactly 5 seconds. If you see that number, this is a strong suspect. Mitigation is `single-request-reopen` in resolv.conf options, or disabling AAAA lookups if you have no IPv6. ## Diagnosing properly Get a shell with tools in the pod's network namespace: ```bash kubectl debug -it mypod --image=nicolaka/netshoot --target=app ``` That shares the network namespace, so you see [exactly what the application sees](https://debugly.dev/debug-distroless-container-no-shell/). Then: ```bash cat /etc/resolv.conf # resolve the way the app does getent hosts postgres # see every query the resolver actually makes strace -f -e trace=network getent hosts api.payments.com 2>&1 | grep -i sendto # watch DNS traffic tcpdump -i any -n port 53 -c 30 # query the cluster DNS directly, bypassing search logic dig @10.96.0.10 postgres.default.svc.cluster.local +short ``` The `strace` line is the one that settles arguments. It shows the actual sequence of queries including every search domain attempt, and it makes the amplification visible. Same technique as [using strace to find where time goes generally](https://debugly.dev/strace-for-web-developers/). To compare tools directly: ```bash dig +search postgres # honours the search list dig postgres # does not ``` If `dig +search` works and plain `dig` does not, you are relying on the search path, which is fine within a namespace and fragile across one. ## What actually fixed it ### Lower ndots for services calling external hosts ```yaml spec: dnsConfig: options: - name: ndots value: "2" ``` Now `api.payments.com` is tried as absolute first. One query instead of five. Cluster internal short names still resolve through the search path. This is the single highest value change for a service that mostly calls external APIs, and it typically cuts DNS query volume by 70 to 80 percent. ### Fully qualify external hostnames A trailing dot makes a name absolute regardless of ndots: ``` https://api.payments.com./v2/charge ``` Ugly and better supported than you would expect. Worth doing for your highest volume external endpoints even if you also set ndots. ### NodeLocal DNSCache A caching resolver as a DaemonSet on every node. Pods query a local address instead of crossing the network to CoreDNS, and it upgrades the upstream to TCP, which eliminates the silent-UDP-drop-plus-timeout failure entirely. This is the most valuable infrastructure level fix and it is the one I would do first on any cluster with DNS problems. ### Shorten the resolver timeout ```yaml dnsConfig: options: - name: timeout value: "1" - name: attempts value: "3" ``` Three fast attempts beat one five second stall. It converts a catastrophic tail into a mild one, and it is a backstop rather than a fix. ### Scale and monitor CoreDNS ```bash kubectl -n kube-system get deploy coredns ``` Two replicas is the default and it is frequently not enough. Watch request duration, error rate, and cache hit ratio. CoreDNS is infrastructure every single pod depends on and it is routinely unmonitored, which is [the shared infrastructure blind spot](https://debugly.dev/three-metrics-that-catch-incidents/) in a nutshell. ## Alpine and musl Worth a separate note because the behaviour differs. musl's resolver historically did not implement the search domain logic the same way, did not support more than a few nameservers, and handled the `ndots` option differently. Several versions had no `search` support at all in some paths. The practical consequence: a service that resolves fine on a Debian base can fail on Alpine with the same resolv.conf. If you moved to Alpine for image size and DNS started behaving strangely, that is a real and known difference. Options are installing `musl-locales` and a compatibility layer, using fully qualified names everywhere so search is irrelevant, or moving to a distroless or slim Debian base. ## A checklist - Test with `getent hosts`, not `nslookup` - Read `/etc/resolv.conf` in the actual pod - Count the queries with strace before assuming DNS is fine - Check whether the name crosses a namespace boundary - Look for exactly-5-second latencies, which point at resolver timeouts - Check CoreDNS CPU and error rate, not just whether it is running - Confirm whether your base image is glibc or musl --- Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)