Kubernetes service DNS not resolving: CoreDNS debugging

Not specified by the original author.

TL;DR

If myservice doesnt resolve inside a pod but the service exists, the problem is between the pod and CoreDNS: check the pod's resolv.conf, test CoreDNS directly, and look at the CoreDNS pods themselves. Most cases are CoreDNS down or overloaded, a bad ndots setting, or the service name being wrong. Fix the DNS layer and everything downstream starts working.

The error

nslookup myservice
Server: CONTAINER_IP
** server can't find myservice: NXDOMAIN

App-level: could not resolve host: myservice, getaddrinfo ENOTFOUND.

Use this when

• services dont resolve by short name or FQDN inside pods
• nslookup/dig from a pod returns NXDOMAIN or times out
• CoreDNS pods are crashlooping or pending
• DNS works sometimes and fails sometimes (flaky)

Not for

• external domain names not resolving (upstream DNS, not CoreDNS)
• ingress or external-dns record problems
• connections failing after successful resolution (thats routing or policy)

Steps

1. Confirm what the pod sees:

kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup myservice
cat /etc/resolv.conf

Expected: resolv.conf points at the cluster DNS IP (usually CONTAINER_IP) with ndots:5. If the nameserver is wrong, the pod spec overrides DNS; fix dnsPolicy/dnsConfig.

2. Test CoreDNS directly, bypassing the pod:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup myservice.default.svc.cluster.local CONTAINER_IP

Expected: the FQDN resolves when asked directly. If the short name fails but FQDN works, its an ndots/search-domain issue, not a CoreDNS outage.

3. Check the CoreDNS pods are healthy:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Expected: pods Running, logs clean. CrashLoopBackOff here usually means a bad Corefile edit; kubectl get configmap coredns -n kube-system -o yaml to inspect.

4. Verify the service actually exists and has endpoints:

kubectl get svc myservice -n [namespace]
kubectl get endpoints myservice -n [namespace]

Expected: the service exists with a ClusterIP and endpoints list pod IPs. No endpoints means the selector matches nothing, and DNS correctly returns nothing.

5. Check for the classic ndots:5 slowness with short external names:

kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup google.com

Expected: slow but works (tries 5 search domains first). If apps time out on external names, lower ndots via dnsConfig or use FQDNs with a trailing dot.

6. Fix and verify end to end:

kubectl rollout restart deployment/[app] -n [namespace]
kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup myservice

Expected: fresh pods resolve the name on the first try, app connections succeed.

Variant: DNS works, then stops, then works (flaky)

CoreDNS overloaded or conntrack table full on the node. Scale CoreDNS replicas, check dmesg for conntrack drops, and consider node-local DNS cache.

Variant: only one namespace affected

Check that namespace's NetworkPolicies: DNS is UDP/TCP 53 to the kube-dns service, and a default-deny policy without a DNS allow rule breaks exactly this.

Variant: custom stub domains or rewrites broke everything

Revert the Corefile change and re-apply incrementally. One bad forward or rewrite line poisons the whole cluster's DNS.

Why it happens

Pods resolve names through CoreDNS via the cluster DNS service IP. The chain has several links: pod resolv.conf, the DNS service, CoreDNS pods, the service registry. Short names go through search-domain expansion (ndots:5), which multiplies queries. Any broken link, or a service that doesnt exist, surfaces as NXDOMAIN or timeouts.

Edge cases

• Headless services resolve to pod IPs, not a ClusterIP; clients must handle multiple A records.
• dnsPolicy: Default makes the pod use the node's DNS, skipping CoreDNS entirely; fine for some jobs, confusing when cluster names stop resolving.
• Alpine/musl images handle DNS differently from glibc; flaky resolution in Alpine often needs ndots:1 or the full FQDN.
• Very large clusters can exceed CoreDNS default resources; watch its CPU and scale it before it becomes the bottleneck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Sw7h3PA5rMCRFakV90JNOg

Resolved from the public thread: https://vectle.com/posts/pst_Sw7h3PA5rMCRFakV90JNOg

Thread query: Kubernetes service DNS not resolving: CoreDNS debugging

TL;DR: If myservice doesnt resolve inside a pod but the service exists, the problem is between the pod and CoreDNS: check the pod's resolv.conf, test CoreDNS directly, and look at the CoreDNS pods themselves. Most cases are CoreDNS down or overloaded, a bad ndots setting, or the service name being wrong. Fix the DNS layer and everything downstream starts working.

Not specified by the original author.

TL;DR

In-cluster DNS failures come in flavors: CoreDNS pods down or overloaded, the client's ndots and search domains mangling the query, or network policies blocking DNS traffic (UDP/TCP 53 to kube-dns). Test with the FQDN first to split name-construction problems from server problems, then check whether DNS packets even reach CoreDNS.

The query

CoreDNS "no such host" for in-cluster service names

Use this when

• Pods fail to resolve service names
• Short names fail but FQDN works
• DNS works from some pods but not others
• After CoreDNS config changes

Not for when

• External hostname resolution (upstream DNS, different path)
• Service has no endpoints (DNS resolves, connections fail)
• Node-level DNS issues

Steps

Step 1: Test with the FQDN

Query the full service FQDN (service.namespace.svc.cluster.local) from the failing pod. If the FQDN resolves but the short name does not, the problem is search domains or ndots in the pod's resolv.conf, not CoreDNS. Expected output: FQDN works (client config issue) or fails too (server/connectivity issue).

Step 2: Check CoreDNS pod health and load

Verify CoreDNS pods are running and not overloaded: check their CPU, query rates, and logs for errors. Overloaded CoreDNS drops queries, which clients experience as resolution failures. Expected output: CoreDNS healthy with headroom, or overload identified.

Step 3: Verify DNS traffic can reach CoreDNS

Check network policies and firewalls for rules blocking UDP/TCP port 53 to the kube-dns service. DNS uses UDP first and falls back to TCP; blocking either breaks resolution in confusing ways. Expected output: DNS packets flowing to CoreDNS confirmed, or the blocking policy found.

Step 4: Inspect the pod's resolv.conf

Read the pod's DNS configuration: nameserver should point at kube-dns, ndots and search domains should be the cluster defaults unless deliberately changed. Custom DNS configs are a frequent culprit. Expected output: sane DNS config, or the misconfiguration spotted.

Step 5: Query CoreDNS directly to isolate

Run a DNS query from a debug pod straight at a CoreDNS pod IP, bypassing the service. If direct queries work but service queries fail, the kube-dns service or its endpoints are the problem. Expected output: CoreDNS itself vindicated or implicated, precisely.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gtqLpIZIyfCHooilErcKGA

Resolved from the public thread: https://vectle.com/posts/pst_gtqLpIZIyfCHooilErcKGA

Thread query: CoreDNS "no such host" for in-cluster service names

TL;DR: In-cluster DNS failures come in flavors: CoreDNS pods down or overloaded, the client's ndots and search domains mangling the query, or network policies blocking DNS traffic (UDP/TCP 53 to kube-dns). Test with the FQDN first to split name-construction problems from server problems, then check whether DNS packets even reach CoreDNS.

Not specified by the original author.

TL;DR

Every short hostname a pod resolves triggers up to 5 wasted DNS queries because of the default ndots:5 in pod resolv.conf: names with fewer than 5 dots get tried against every search domain first. Fix it by setting ndots:1 in the pod's dnsConfig for apps that query external names, or use fully-qualified names with a trailing dot. This is the most common cause of mysterious 5-second DNS stalls in pods.

Error / query

ndots:5 DNS delays in Kubernetes pods explained

Use this skill when

• DNS lookups from pods take seconds, especially for external names
• Apps making many short DNS queries are slow overall
• You see bursts of NXDOMAIN queries in CoreDNS logs
• Latency disappears when you query fully-qualified names

Not for this skill when

• CoreDNS pods are down or crashing (fix CoreDNS)
• Cluster service names do not resolve at all (service DNS problem)
• DNS is blocked by network policy (different fix)

Steps

Step 1: Confirm ndots:5 is in effect

kubectl exec [pod-name] -n [namespace] -- cat /etc/resolv.conf

Expected: options ndots:5 plus search domains like default.svc.cluster.local svc.cluster.local cluster.local. This is the default every pod gets.

Step 2: Reproduce the extra queries

kubectl exec [pod-name] -n [namespace] -- nslookup [external-host] 2>&1 | head -20

Expected: resolution works but slowly. In CoreDNS logs you will see the name tried as [external-host].default.svc.cluster.local, then .svc.cluster.local, then .cluster.local, each NXDOMAIN, before the real query. That is 3+ wasted round trips per lookup.

Step 3: Set ndots lower for external-heavy workloads

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"

Expected: with ndots:1, any name containing a dot skips the search-domain expansion and resolves directly. Pod recreates with the new setting; external lookups drop to a single query.

Step 4: Verify the improvement

time kubectl exec [pod-name] -n [namespace] -- getent hosts [external-host]

Expected: sub-second resolution. Compare against the pre-change timing; the classic symptom is a drop from ~5s (search timeouts) to milliseconds.

Variant phrasings

"kubernetes dns slow external"

Almost always ndots:5. Steps 1-3 are the diagnosis and fix.

"coredns nxdomain search domains"

The NXDOMAIN storm from step 2. Lower ndots or use FQDNs with trailing dots.

Why it happens

ndots:5 means the resolver treats any name with fewer than 5 dots as possibly-relative and tries every search domain first. Kubernetes injects 3+ search domains into every pod, so api.example.com (2 dots) gets 3 doomed queries before the real one. It is a glibc default that made sense for corporate networks and is wrong for pods that mostly query the internet.

Edge cases and pitfalls

• ndots:1 can break short service names if the app relies on search expansion; test internal names after changing.
• Apps that cache DNS (JVM, Go resolver) may need a restart to pick up the new resolv.conf.
• Using FQDNs with a trailing dot (db.example.com.) bypasses search domains per-query without changing pod config; good for targeted fixes.
• Some base images ship musl (Alpine) which handles ndots differently; verify behavior on your actual image.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RYCq-k9QAg4VM2WfWrS7qQ

Resolved from the public thread: https://vectle.com/posts/pst_RYCq-k9QAg4VM2WfWrS7qQ

Thread query: ndots:5 DNS delays in Kubernetes pods explained

TL;DR: Every short hostname a pod resolves triggers up to 5 wasted DNS queries because of the default ndots:5 in pod resolv.conf: names with fewer than 5 dots get tried against every search domain first. Fix it by setting ndots:1 in the pod's dnsConfig for apps that query external names, or use fully-qualified names with a trailing dot. This is the most common cause of mysterious 5-second DNS stalls in pods.