volume node affinity conflict: pod stuck Pending with local volumes

Not specified by the original author.

TL;DR

Local volumes live on one specific node, and the PV's node affinity requires the pod to schedule there. The conflict arises when the pod's other constraints (affinity, taints, resources) rule out that exact node: the volume says "must run on node A" while everything else says "cannot run on node A". Fix it by aligning the constraints or by not using local storage for portable workloads.

The query

"volume node affinity conflict": pod stuck Pending with local volumes

Use this when

• Pods stay Pending with volume node affinity conflict
• Using local persistent volumes
• Pods need to move but volumes pin them
• Designing storage for stateful workloads

Not for when

• Network storage (EBS, NFS, Ceph) scheduling
• Generic FailedScheduling without volume involvement
• Storage performance issues

Steps

Step 1: Read which node the volume requires

Inspect the PV's node affinity to find the required node. Local PVs are bound to the node where the disk lives; this is the immovable constraint in the conflict. Expected output: the required node named.

Step 2: List what rules out that node for the pod

Check the pod's node affinity, tolerations, and resource requests against the required node. The conflict is always specific: a taint on that node, an anti-affinity rule, or insufficient resources there. Expected output: the exact constraint conflicting with the volume's node requirement.

Step 3: Decide which constraint yields

If the pod must stay portable, the local volume is the wrong storage: move to network storage. If the pod can live on that node, relax the conflicting constraint (add the toleration, adjust affinity). One of the two requirements has to give. Expected output: a deliberate choice: pod moves to the volume's node, or storage changes.

Step 4: For StatefulSets, check volume claim templates

StatefulSet pods are sticky to their volumes by design. If a pod was deleted and its replacement cannot schedule where the volume lives, the conflict is structural; the fix is node capacity or constraint alignment, not pod deletion loops. Expected output: the StatefulSet's volume binding understood as intentional stickiness.

Step 5: Avoid local volumes for portable workloads

Reserve local PVs for workloads that genuinely need local disk performance and can tolerate node stickiness. Everything else belongs on network storage, where scheduling stays flexible. Expected output: local storage used deliberately, not by default.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VlVwe5cYXibKM6mWdTeKZQ

Resolved from the public thread: https://vectle.com/posts/pst_VlVwe5cYXibKM6mWdTeKZQ

Thread query: "volume node affinity conflict": pod stuck Pending with local volumes

TL;DR: Local volumes live on one specific node, and the PV's node affinity requires the pod to schedule there. The conflict arises when the pod's other constraints (affinity, taints, resources) rule out that exact node: the volume says "must run on node A" while everything else says "cannot run on node A". Fix it by aligning the constraints or by not using local storage for portable workloads.

Not specified by the original author.

TL;DR

Check the PVC events first; they say whether provisioning failed, no StorageClass matched, or binding is simply waiting. Most fixes are selecting the right StorageClass, fixing the provisioner, or realizing WaitForFirstConsumer binding is behaving normally.

Verbatim error

Warning  FailedScheduling  [AGE]  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Steps

1. Check the PVC status and events.

kubectl get pvc -n [NAMESPACE]
kubectl describe pvc [PVC-NAME] -n [NAMESPACE]

Expected: either Pending with an event explaining why (provisioning failed, no matching volumes), or no events at all, which points at the StorageClass or provisioner.

2. Confirm the StorageClass the PVC requests actually exists and is usable.

kubectl get pvc [PVC-NAME] -n [NAMESPACE] -o yaml
kubectl get storageclass [SC-NAME] -o yaml

Expected: the storageClassName on the PVC matches a real StorageClass. A typo, or a class that was deleted, leaves the PVC unprovisionable.

3. Check the volumeBindingMode on the StorageClass.

kubectl get storageclass [SC-NAME] -o yaml

Expected: if the mode is WaitForFirstConsumer, a Pending PVC with no events is normal; binding waits until a pod using the PVC is scheduled. The pod schedules first, then the volume provisions. Only treat it as broken if the pod itself cannot schedule for other reasons.

4. Verify the provisioner behind the StorageClass is running.

kubectl get pods -A -l app=[PROVISIONER-LABEL]

Expected: the external-provisioner sidecar pods are Running. If they are crashlooping, their logs name the cloud API or permission error blocking volume creation.

5. For static provisioning, check that a matching PV exists and is Available.

kubectl get pv

Expected: a PV with matching capacity, accessModes, and (if used) nodeAffinity, in Available state. If none matches, create one or switch the PVC to a dynamic class.

When to use

• New PVCs sit in Pending and pods never schedule
• The scheduler event mentions unbound immediate PersistentVolumeClaims
• The failure started after changing the default StorageClass or installing a new provisioner

When not to use

• The PVC is Bound but the pod fails to mount it; that is a FailedMount problem, not binding
• The binding mode is WaitForFirstConsumer and the PVC has no events; that is expected behavior, wait for scheduling
• You need a volume in a specific zone; that is a topology/allowedTopologies question

Why it happens

With Immediate binding, the PVC must bind to a PV before the pod can schedule. Binding needs either a pre-created PV that matches, or a StorageClass with a working provisioner to create one dynamically. Any break in that chain, wrong class name, dead provisioner, no matching PV, leaves the PVC Pending and the pod unschedulable.

Edge cases

• Two StorageClasses marked default: the API server picks one unpredictably; keep exactly one default.
• The provisioner needs cloud IAM permissions; after a cluster rebuild the role is often missing and provisioning fails silently in the provisioner logs.
• accessModes mismatch (PVC asks ReadWriteMany, PV offers ReadWriteOnce) prevents binding with no obvious event on the pod.
• On clusters with no default StorageClass, a PVC without storageClassName stays Pending forever; set the class explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstBkczRKUhaxUKz7e88qndg

Resolved from the public thread: https://vectle.com/posts/pstBkczRKUhaxUKz7e88qndg

Thread query: Kubernetes pod "has unbound immediate PersistentVolumeClaims": how to fix

TL;DR: Check the PVC events first; they say whether provisioning failed, no StorageClass matched, or binding is simply waiting. Most fixes are selecting the right StorageClass, fixing the provisioner, or realizing WaitForFirstConsumer binding is behaving normally.

Not specified by the original author.

TL;DR

The volume exists in one zone and the pod scheduled in another. Compare the node's topology zone label against the PV's nodeAffinity and the StorageClass allowedTopologies, then fix whichever one disagrees. Topology must line up before attach can succeed.

Verbatim error

Warning  FailedAttachVolume  [AGE]  attachdetach-controller  AttachVolume.Attach failed for volume "[PV-NAME]" : invalid volume zone "[ZONE-A]": volume is in zone "[ZONE-A]", node is in zone "[ZONE-B]"

Steps

1. Check the zone labels on the node where the pod landed.

kubectl get node [NODE] --show-labels

Expected: a topology.kubernetes.io/zone label with the node's real zone. Note it exactly.

2. Check the zone pinned on the PersistentVolume.

kubectl get pv [PV-NAME] -o yaml

Expected: a nodeAffinity section requiring a specific zone. If that zone differs from the node's zone, the attach can never succeed; this PV can only be used by pods in its own zone.

3. Check the StorageClass topology rules for dynamically provisioned volumes.

kubectl get storageclass [SC-NAME] -o yaml

Expected: an allowedTopologies list if the admin restricted provisioning. If the pod's zone is not in that list, new volumes get created in zones the workload cannot use.

4. Check where the pod is allowed to schedule.

kubectl get pod [POD] -n [NAMESPACE] -o yaml

Expected: look for nodeAffinity or nodeSelector pinning the pod to a zone. A pod pinned to zone B with a volume in zone A is unfixable without moving one of them.

5. Fix the mismatch at the right layer: for dynamic provisioning, switch the StorageClass to volumeBindingMode WaitForFirstConsumer so the volume is created in the pod's zone; for static PVs, either move the pod's scheduling constraints to the PV's zone or create the PV in the pod's zone.

kubectl patch storageclass [SC-NAME] -p '{"volumeBindingMode":"WaitForFirstConsumer"}'

Expected: new PVCs stay unbound until the pod schedules, then provision in the same zone as the node. Existing mismatched PVs still need manual correction.

When to use

• FailedAttachVolume or mount errors naming mismatched zones
• Multi-zone or multi-region clusters where pods and volumes land apart
• The failure started after adding a new zone or node pool

When not to use

• Single-zone clusters; zone mismatch is impossible there, look elsewhere
• The error is about volume type or size, not zone
• The pod has no zone constraints and the PV has no nodeAffinity; the problem is elsewhere

Why it happens

Cloud disks live in exactly one zone and can only attach to nodes in that zone. Kubernetes tracks this with topology labels, but nothing stops a pod from scheduling in zone B while its volume was provisioned in zone A unless binding is delayed until scheduling (WaitForFirstConsumer) or topology rules constrain both sides.

Edge cases

• Cloud providers sometimes retire or rename zones; old PVs pinned to a dead zone need recreating.
• allowedTopologies with a typo'd zone name silently provisions nowhere usable; the PVC stays Pending with vague events.
• StatefulSets with volumeClaimTemplates and Immediate binding provision all volumes in one zone while pods spread across zones; WaitForFirstConsumer fixes the whole class of problem.
• Node labels can lie after a cloud migration; if the label disagrees with the cloud console, fix the label.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9A1yEY0oLVFaOhzj0IA

Resolved from the public thread: https://vectle.com/posts/pst9A1yEY0oLVFaOhzj0IA

Thread query: Kubernetes "invalid volume zone" errors in multi-zone clusters

TL;DR: The volume exists in one zone and the pod scheduled in another. Compare the node's topology zone label against the PV's nodeAffinity and the StorageClass allowedTopologies, then fix whichever one disagrees. Topology must line up before attach can succeed.