Kubernetes network policy blocking traffic: how to test

TL;DR

If pod-to-pod traffic dies right after a NetworkPolicy lands, test it directly: run a temporary client pod and curl the target, then read the policies in both namespaces. NetworkPolicies are additive and default-deny is silent, so the fix is usually adding the missing ingress or egress rule, or the missing podSelector label.

The query

Kubernetes network policy blocking traffic: how to test

Symptom: connections time out between pods that used to talk fine.

Use this when

• pod-to-pod connections time out after a policy change
• DNS resolves but TCP/HTTP hangs
• a new default-deny policy just went in
• you need to prove the policy is (or isnt) the cause

Not for

• DNS failures (resolve first, then test traffic)
• services with no endpoints (nothing to connect to)
• node-level firewall or CNI breakage (affects everything, not policy-shaped)

Steps

1. Prove the failure with a test client pod:

kubectl run nettest --rm -it --image=busybox --restart=Never -n [client-ns] -- wget -qO- --timeout=5 http://target-svc.target-ns:8080/

Expected: wget: download timed out confirms the block. Success here means the policy isnt the problem, look elsewhere.

1. List the policies in both namespaces:

kubectl get networkpolicy -n [client-ns]
kubectl get networkpolicy -n [target-ns]

Expected: note every policy and its podSelector. An empty selector ({}) selects ALL pods in the namespace, which surprises people.

1. Read the actual rules:

kubectl get networkpolicy [policy-name] -n [target-ns] -o yaml

Expected: check policyTypes (Ingress, Egress), the podSelector matchLabels, and whether the from/to blocks include your client's labels and namespace. The common miss: the rule allows the right pods but the client's labels dont match.

1. Check for default-deny:

kubectl get networkpolicy -n [target-ns] -o yaml | grep -B 3 -A 3 podSelector: {}

Expected: a policy selecting all pods with no ingress rules denies all ingress. Thats the design, but every legitimate flow then needs an explicit allow.

1. Dont forget DNS: a default-deny egress policy kills DNS too:

egress:
  - to:
      - namespaceSelector:
          matchLabels:
            name: kube-system
        podSelector:
          matchLabels:
            k8s-app: kube-dns
    ports:
      - protocol: UDP
        port: 53

Expected: with this rule, pods can resolve names again. DNS-blocked pods fail in confusing ways that look like app errors.

1. Apply the fix and re-run the test from step 1:

kubectl apply -f networkpolicy.yaml
kubectl run nettest --rm -it --image=busybox --restart=Never -n [client-ns] -- wget -qO- --timeout=5 http://target-svc.target-ns:8080/

Expected: the wget succeeds. If it still times out, the policy wasnt the (only) cause.

Variant: works pod-to-pod but not from outside the cluster

Ingress policies dont cover ingress controllers specially; the controller pods need explicit allow rules, or the traffic never reaches the backend.

Variant: policy looks right but traffic still blocked

Labels are the usual lie: kubectl get pod --show-labels on both ends and compare against the policy selectors character by character. Also check the CNI actually enforces policies (some CNIs need a flag).

Variant: intermittent blocking

Rolling pods get new IPs; policies based on ipBlock CIDRs go stale. Prefer podSelector and namespaceSelector over IP blocks.

Why it happens

NetworkPolicies are a whitelist: once any policy selects a pod, everything not explicitly allowed is denied, silently. People add a default-deny for security, forget one flow (DNS is the classic), and spend hours debugging the app when the network was the wall. The test-client-pod approach proves it in one command.

Edge cases

• Egress policies affect the pod's own DNS lookups; always allow kube-dns when you lock down egress.
• Policies are namespace-scoped; cross-namespace traffic needs namespaceSelector rules on both sides as appropriate.
• Some managed CNIs log policy drops; check CNI logs for definitive proof.
• hostNetwork pods bypass NetworkPolicy entirely, which can confuse tests; dont use them as the test client.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fiixGHNIlLIeSFG7rVT6TQ