Kubernetes service DNS not resolving: CoreDNS debugging

TL;DR

If myservice doesnt resolve inside a pod but the service exists, the problem is between the pod and CoreDNS: check the pod's resolv.conf, test CoreDNS directly, and look at the CoreDNS pods themselves. Most cases are CoreDNS down or overloaded, a bad ndots setting, or the service name being wrong. Fix the DNS layer and everything downstream starts working.

The error

nslookup myservice
Server: CONTAINER_IP
** server can't find myservice: NXDOMAIN

App-level: could not resolve host: myservice, getaddrinfo ENOTFOUND.

Use this when

• services dont resolve by short name or FQDN inside pods
• nslookup/dig from a pod returns NXDOMAIN or times out
• CoreDNS pods are crashlooping or pending
• DNS works sometimes and fails sometimes (flaky)

Not for

• external domain names not resolving (upstream DNS, not CoreDNS)
• ingress or external-dns record problems
• connections failing after successful resolution (thats routing or policy)

Steps

1. Confirm what the pod sees:

kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup myservice
cat /etc/resolv.conf

Expected: resolv.conf points at the cluster DNS IP (usually CONTAINER_IP) with ndots:5. If the nameserver is wrong, the pod spec overrides DNS; fix dnsPolicy/dnsConfig.

1. Test CoreDNS directly, bypassing the pod:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup myservice.default.svc.cluster.local CONTAINER_IP

Expected: the FQDN resolves when asked directly. If the short name fails but FQDN works, its an ndots/search-domain issue, not a CoreDNS outage.

1. Check the CoreDNS pods are healthy:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Expected: pods Running, logs clean. CrashLoopBackOff here usually means a bad Corefile edit; kubectl get configmap coredns -n kube-system -o yaml to inspect.

1. Verify the service actually exists and has endpoints:

kubectl get svc myservice -n [namespace]
kubectl get endpoints myservice -n [namespace]

Expected: the service exists with a ClusterIP and endpoints list pod IPs. No endpoints means the selector matches nothing, and DNS correctly returns nothing.

1. Check for the classic ndots:5 slowness with short external names:

kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup google.com

Expected: slow but works (tries 5 search domains first). If apps time out on external names, lower ndots via dnsConfig or use FQDNs with a trailing dot.

1. Fix and verify end to end:

kubectl rollout restart deployment/[app] -n [namespace]
kubectl run dns-test --rm -it --image=busybox --restart=Never -- nslookup myservice

Expected: fresh pods resolve the name on the first try, app connections succeed.

Variant: DNS works, then stops, then works (flaky)

CoreDNS overloaded or conntrack table full on the node. Scale CoreDNS replicas, check dmesg for conntrack drops, and consider node-local DNS cache.

Variant: only one namespace affected

Check that namespace's NetworkPolicies: DNS is UDP/TCP 53 to the kube-dns service, and a default-deny policy without a DNS allow rule breaks exactly this.

Variant: custom stub domains or rewrites broke everything

Revert the Corefile change and re-apply incrementally. One bad forward or rewrite line poisons the whole cluster's DNS.

Why it happens

Pods resolve names through CoreDNS via the cluster DNS service IP. The chain has several links: pod resolv.conf, the DNS service, CoreDNS pods, the service registry. Short names go through search-domain expansion (ndots:5), which multiplies queries. Any broken link, or a service that doesnt exist, surfaces as NXDOMAIN or timeouts.

Edge cases

• Headless services resolve to pod IPs, not a ClusterIP; clients must handle multiple A records.
• dnsPolicy: Default makes the pod use the node's DNS, skipping CoreDNS entirely; fine for some jobs, confusing when cluster names stop resolving.
• Alpine/musl images handle DNS differently from glibc; flaky resolution in Alpine often needs ndots:1 or the full FQDN.
• Very large clusters can exceed CoreDNS default resources; watch its CPU and scale it before it becomes the bottleneck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Sw7h3PA5rMCRFakV90JNOg