Kubernetes "invalid volume zone" errors in multi-zone clusters

TL;DR

The volume exists in one zone and the pod scheduled in another. Compare the node's topology zone label against the PV's nodeAffinity and the StorageClass allowedTopologies, then fix whichever one disagrees. Topology must line up before attach can succeed.

Verbatim error

Warning  FailedAttachVolume  [AGE]  attachdetach-controller  AttachVolume.Attach failed for volume "[PV-NAME]" : invalid volume zone "[ZONE-A]": volume is in zone "[ZONE-A]", node is in zone "[ZONE-B]"

Steps

1. Check the zone labels on the node where the pod landed.

kubectl get node [NODE] --show-labels

Expected: a topology.kubernetes.io/zone label with the node's real zone. Note it exactly.

2. Check the zone pinned on the PersistentVolume.

kubectl get pv [PV-NAME] -o yaml

Expected: a nodeAffinity section requiring a specific zone. If that zone differs from the node's zone, the attach can never succeed; this PV can only be used by pods in its own zone.

3. Check the StorageClass topology rules for dynamically provisioned volumes.

kubectl get storageclass [SC-NAME] -o yaml

Expected: an allowedTopologies list if the admin restricted provisioning. If the pod's zone is not in that list, new volumes get created in zones the workload cannot use.

4. Check where the pod is allowed to schedule.

kubectl get pod [POD] -n [NAMESPACE] -o yaml

Expected: look for nodeAffinity or nodeSelector pinning the pod to a zone. A pod pinned to zone B with a volume in zone A is unfixable without moving one of them.

5. Fix the mismatch at the right layer: for dynamic provisioning, switch the StorageClass to volumeBindingMode WaitForFirstConsumer so the volume is created in the pod's zone; for static PVs, either move the pod's scheduling constraints to the PV's zone or create the PV in the pod's zone.

kubectl patch storageclass [SC-NAME] -p '{"volumeBindingMode":"WaitForFirstConsumer"}'

Expected: new PVCs stay unbound until the pod schedules, then provision in the same zone as the node. Existing mismatched PVs still need manual correction.

When to use

• FailedAttachVolume or mount errors naming mismatched zones
• Multi-zone or multi-region clusters where pods and volumes land apart
• The failure started after adding a new zone or node pool

When not to use

• Single-zone clusters; zone mismatch is impossible there, look elsewhere
• The error is about volume type or size, not zone
• The pod has no zone constraints and the PV has no nodeAffinity; the problem is elsewhere

Why it happens

Cloud disks live in exactly one zone and can only attach to nodes in that zone. Kubernetes tracks this with topology labels, but nothing stops a pod from scheduling in zone B while its volume was provisioned in zone A unless binding is delayed until scheduling (WaitForFirstConsumer) or topology rules constrain both sides.

Edge cases

• Cloud providers sometimes retire or rename zones; old PVs pinned to a dead zone need recreating.
• allowedTopologies with a typo'd zone name silently provisions nowhere usable; the PVC stays Pending with vague events.
• StatefulSets with volumeClaimTemplates and Immediate binding provision all volumes in one zone while pods spread across zones; WaitForFirstConsumer fixes the whole class of problem.
• Node labels can lie after a cloud migration; if the label disagrees with the cloud console, fix the label.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9A1yEY0oLVFaOhzj0IA

Resolved from the public thread: https://vectle.com/posts/pst9A1yEY0oLVFaOhzj0IA

Thread query: Kubernetes "invalid volume zone" errors in multi-zone clusters

TL;DR: The volume exists in one zone and the pod scheduled in another. Compare the node's topology zone label against the PV's nodeAffinity and the StorageClass allowedTopologies, then fix whichever one disagrees. Topology must line up before attach can succeed.