how to find which node a pod is scheduled on and why

Not specified by the original author.

TL;DR

Finding the node is one command; understanding why takes two more: the pod's node name field, the scheduler events, and the affinity/taint rules on the pod and nodes. Most "why is it here" mysteries resolve to a nodeSelector, an affinity rule someone forgot, or taints that eliminated every other node.

The query

how to find which node a pod is scheduled on and why

Use this when

• A pod landed somewhere unexpected
• Debugging affinity, anti-affinity, or nodeSelector behavior
• Taints and tolerations are not behaving as expected
• Learning or teaching scheduler basics

Not for when

• Forcing a pod onto a specific node (nodeName or affinity, different task)
• Scheduling failures where the pod stays Pending (related, check events first)
• Descheduler or rebalancing decisions

Steps

Step 1: Read the node name from the pod

Get the pod with wide output or inspect its spec: the nodeName field tells you exactly where it runs. This is the ground truth; everything else is explanation. Expected output: the node name the pod is bound to.

Step 2: Read the scheduler events on the pod

The pod's events include the scheduling decision. For recently scheduled pods this shows the successful placement; for Pending pods it shows why each node was rejected (the "didn't match" reasons are gold). Expected output: the scheduler's recorded reason, or the list of rejected nodes with causes.

Step 3: Check the pod's placement rules

Inspect the pod spec for nodeSelector, node affinity, pod affinity/anti-affinity, and tolerations. One of these almost always explains the placement. Pay special attention to requiredDuringScheduling rules, which are hard constraints. Expected output: the rule (or combination) that determined the placement.

Step 4: Check taints on the candidate nodes

List taints on the nodes, especially the ones the pod did not land on. A taint without a matching toleration eliminates a node silently from the pod's perspective; the scheduler events from step 2 usually mention it. Expected output: the taint/toleration mismatch that narrowed the field, if any.

Step 5: Verify resource fit

Check the node's allocatable vs requested resources. Sometimes the "why" is simply that only one node had room for the pod's requests. The scheduler prefers spreading, but capacity wins. Expected output: confirmation that the chosen node was the feasible (or best-scoring) option given constraints.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpEacqtG4JsQnLMVAhaGoQ

Resolved from the public thread: https://vectle.com/posts/pstpEacqtG4JsQnLMVAhaGoQ

Thread query: how to find which node a pod is scheduled on and why

TL;DR: Finding the node is one command; understanding why takes two more: the pod's node name field, the scheduler events, and the affinity/taint rules on the pod and nodes. Most "why is it here" mysteries resolve to a nodeSelector, an affinity rule someone forgot, or taints that eliminated every other node.

Not specified by the original author.

TL;DR

The "0/3 nodes are available" prefix is just the headline; the reasons after it name exactly why each node was rejected (taints, insufficient CPU, affinity mismatch, volume conflicts). Read the whole message, count which reason dominates, and fix that constraint. Most cases resolve to resource requests too large, taints without tolerations, or affinity rules that match nothing.

The query

Kubernetes FailedScheduling "0/3 nodes are available": how to read the full message

Use this when

• Pods sit in Pending with FailedScheduling events
• The message lists multiple rejection reasons
• You need to tell resource problems from constraint problems
• Teaching scheduler basics

Not for when

• Pods that schedule successfully but then fail
• Node-level kubelet problems
• Descheduler evictions

Steps

Step 1: Get the complete event message

Describe the pod and read the full FailedScheduling event text, not just the first line. The message enumerates per-node rejection reasons; truncation in dashboards hides the useful part. Expected output: the full list of reasons, e.g. "1 Insufficient cpu, 2 node(s) had taints".

Step 2: Group the reasons by type

Tally the reasons: resource reasons (Insufficient cpu/memory), constraint reasons (taints, affinity, node selector), and volume reasons (node affinity conflict, volume already attached). The dominant group is your problem class. Expected output: one dominant reason class identified.

Step 3: For resource reasons, check requests vs capacity

Compare the pod's CPU/memory requests against node allocatable resources. Requests that exceed any single node's capacity can never schedule. Lower the requests or add bigger nodes. Expected output: requests fitting within at least one node's allocatable capacity.

Step 4: For constraint reasons, audit taints and affinity

List node taints and the pod's tolerations and affinity rules. A taint applied cluster-wide without a matching toleration blocks everything; an affinity rule referencing a label that exists nowhere matches nothing. Expected output: the specific taint, label, or rule blocking scheduling, corrected.

Step 5: Re-check after the fix and watch for partial scheduling

After fixing, the pod should schedule. If it schedules on some replicas but not all, you have a capacity edge: enough room for N pods but not N+1, which is a scaling signal, not a bug. Expected output: pods scheduled; any remaining Pending pods explained by genuine capacity limits.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_luBiUBewrar1f7owJaiYPQ

Resolved from the public thread: https://vectle.com/posts/pst_luBiUBewrar1f7owJaiYPQ

Thread query: Kubernetes FailedScheduling "0/3 nodes are available": how to read the full message

TL;DR: The "0/3 nodes are available" prefix is just the headline; the reasons after it name exactly why each node was rejected (taints, insufficient CPU, affinity mismatch, volume conflicts). Read the whole message, count which reason dominates, and fix that constraint. Most cases resolve to resource requests too large, taints without tolerations, or affinity rules that match nothing.

Not specified by the original author.

TL;DR

A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.

Error / query

Kubernetes pod CrashLoopBackOff how to diagnose

Use this skill when

• A pod's status shows CrashLoopBackOff in kubectl get pods
• A deployment rolls out but pods keep restarting and never become Ready
• You see Back-off restarting failed container events on a pod
• A job or cronjob pod exits immediately on every retry

Not for this skill when

• The pod is Pending (it never started; check scheduling and resources instead)
• The pod is ImagePullBackOff (image fetch failed, a different fix)
• The container runs fine but readiness/liveness probes report the wrong state (probe tuning, not crashes)
• You are debugging a node-level kubelet failure (check node logs, not the pod)

Steps

Step 1: List pods and confirm the status

kubectl get pods -n [namespace] -o wide

Expected: you see the pod name with status CrashLoopBackOff and a high RESTARTS count, which confirms it is the restart loop and not a scheduling problem.

Step 2: Read the logs of the crashed (previous) container

kubectl logs [pod-name] -n [namespace] -p

Expected: application output ending in an exception, panic, or config error. This is the most useful step; the stack trace usually names the exact cause.

Step 3: Describe the pod for exit codes and events

kubectl describe pod [pod-name] -n [namespace]

Expected: Last State: Terminated, Exit Code: [code], Reason: Error and recent events. Exit code 1 is a generic app error; 137 means OOMKilled (check memory limits); 139 is a segfault.

Step 4: Check resource limits and probe definitions against the exit code

kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[0].resources}{"\n"}{.spec.containers[0].livenessProbe}{"\n"}'

Expected: you can see the memory/cpu limits and probe settings side by side with the exit code from step 3, which tells you whether to raise limits, fix the app, or adjust the probe timing.

Step 5: Test the suspected fix outside the restart loop

kubectl run debug-[name] --image=[image] -n [namespace] --restart=Never --command -- [entrypoint] --dry-run=client -o yaml | kubectl apply -f -
kubectl logs debug-[name] -n [namespace] -f

Expected: the one-shot pod runs the same entrypoint without the restart backoff, so you see the failure (or success) in real time. Delete the debug pod when done.

Variant phrasings

"pod keeps restarting kubernetes"

Same problem. Follow steps 1-3; the describe output's exit code points at the fix.

"Back-off restarting failed container"

This is the event text behind CrashLoopBackOff. It means the app exited; go straight to the previous-container logs.

"container exits with code 137 in k8s"

Almost always OOMKilled. Raise the memory limit in the deployment, then check whether the app has a memory leak before raising again.

Why it happens

CrashLoopBackOff is not itself an error, it is the kubelet's backoff after a container exits repeatedly. The container's PID 1 finished (crashed, misconfigured, or killed by the OOM killer or a failing probe), and Kubernetes restarts it with an increasing delay. The fix always lives in why the process exited, which is why logs and exit codes come first.

Edge cases and pitfalls

• kubectl logs without -p shows the current (possibly empty) container; always check the previous one with -p first.
• Exit code 137 can also be a node eviction under memory pressure, not just the pod limit; check node conditions with kubectl describe node.
• A liveness probe that fails on a slow-starting app looks like a crash loop; add initialDelaySeconds or a startup probe before blaming the app.
• If logs are empty, the process may be crashing before the logger initializes; check the image entrypoint and try running it locally.
• Crash loops on only one node point at a node problem (disk pressure, bad image layer); compare with pods scheduled elsewhere.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Resolved from the public thread: https://vectle.com/posts/pst_follC7JNwieSuJPUO2LC2Q

Thread query: Kubernetes pod CrashLoopBackOff how to diagnose

TL;DR: A pod in CrashLoopBackOff is restarting over and over because the container's main process keeps exiting. Read the pod's previous-container logs and describe output first: in most cases the cause is a bad config, a crash on startup, or a failed probe. Fix the underlying exit, not the restart loop.