how to debug ingress 404s: ingress controller checklist

TL;DR

Ingress 404s come from a small set of causes: the rule does not match the host or path you are requesting, the backend service has no healthy endpoints, or the controller never picked up the rule. Debug from the outside in: what did you request, which rule should match, does the backend have endpoints. Most 404s die at step two.

The query

how to debug ingress 404s: ingress controller checklist

Use this when

• Ingress returns 404 but the app works via port-forward
• New ingress rules do not seem to take effect
• Some hosts or paths 404 while others work
• After changing ingress configuration

Not for when

• The application itself returns 404 (bypass ingress to confirm)
• TLS certificate errors (different checklist)
• Ingress controller pod crashes

Steps

Step 1: Reproduce precisely and note host, path, and headers

Record the exact request: hostname, path, HTTP method, and relevant headers. Ingress routing keys off these, and "it 404s" without the exact request is undebuggable. Test with curl showing the request line. Expected output: a reproducible curl command that returns the 404.

Step 2: Check which rule should match

List the ingress resources and compare their host and path rules against your request. Common misses: pathType Prefix vs Exact semantics, missing host entry, or the request hitting the default backend because nothing matched. Expected output: identification of the rule that should match, or confirmation that no rule matches (then fix the rules).

Step 3: Verify the backend service has endpoints

Check that the service named in the rule has ready endpoints. A service with zero endpoints makes the controller return 404 or 503 depending on the implementation. The app can be running but not ready (failing readiness probes). Expected output: endpoints listed and ready, or discovery that the backend is the problem, not the routing.

Step 4: Check the controller actually loaded the config

Look at the ingress controller's logs and generated config for your rule. Syntax errors or invalid backends can cause the controller to skip a rule silently. Some controllers expose the running config for inspection. Expected output: your rule present in the controller's active configuration, or an error explaining why it was skipped.

Step 5: Rule out caching and DNS

Confirm you are hitting the ingress you think you are: check DNS resolution, bypass CDN or caching layers, and test with a fresh connection. Stale DNS or a cached 404 from a CDN mimics a broken ingress rule convincingly. Expected output: the 404 reproduces (or disappears) with caching and DNS eliminated as factors.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bVXSqA8ZCuyiTIixno-qww