Karpenter insufficient capacity / no compatible node: how to respond

Not specified by the original author.

TL;DR

Read the Karpenter controller logs for the exact provisioning failure; it names which requirement eliminated every instance type. Then compare the pending pods' constraints against the NodePool and broaden what is too narrow: zones, instance families, architecture, or limits.

Verbatim error

insufficient capacity: no compatible node found for pod [POD]; [N] instance types excluded by requirements

Steps

1. Get the provisioning failure detail from the controller logs.

kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter --tail=300

Expected: lines describing why provisioning failed for your pods, including which requirements excluded instance types or that the cloud returned insufficient capacity. This names the fix.

2. Inspect the NodePool requirements.

kubectl get nodepools
kubectl describe nodepool [NODEPOOL-NAME]

Expected: the requirements list (instance types, zones, architecture, capacity type). Overly narrow requirements, a single instance family in a single zone, are the usual cause of no compatible node.

3. Check the pending pods' scheduling constraints.

kubectl describe pod [PENDING-POD] -n [NAMESPACE]

Expected: nodeSelector, affinity, tolerations, and resource requests. A pod requesting a GPU or a huge memory amount with a NodePool that excludes those can never provision.

4. Broaden the NodePool or add a second one for the constrained workloads.

kubectl -n karpenter edit nodepool [NODEPOOL-NAME]

Expected: after widening zones, instance families, or capacity types (for example allowing spot alongside on-demand), pending pods start provisioning. Keep at least one NodePool permissive enough for general workloads.

5. Check limits and cloud capacity if requirements look fine.

kubectl describe nodepool [NODEPOOL-NAME]

Expected: the limits section caps total CPU or memory; a hit limit stops all provisioning silently. If limits are fine and the cloud truly lacks capacity, broaden zones or instance families, or wait out the shortage.

When to use

• Karpenter logs insufficient capacity or cannot provision for pending pods
• Pending pods appeared after tightening a NodePool or adding strict affinity
• A new workload type never gets nodes while existing ones do

When not to use

• Pods are pending for storage or taint reasons unrelated to node provisioning
• You run cluster-autoscaler; its failure modes and logs are different
• Nodes provision but never join; that is a bootstrap or networking problem

Why it happens

Karpenter matches each pending pod against NodePools, filters cloud instance types by the combined requirements, and provisions the cheapest fit. If the filters exclude everything, or the cloud has no capacity for what remains, provisioning fails and the pod stays pending. The requirements are almost always stricter than intended.

Edge cases

• Disruption budgets and consolidation can look like provisioning failures; check whether Karpenter is churning nodes rather than failing to create them.
• Spot-only NodePools fail during spot shortages with no fallback; keep an on-demand path for critical workloads.
• Requirements inherited from the EC2NodeClass (AMI, subnets, security groups) also filter; a bad subnet mapping excludes whole zones.
• Very large resource requests can exceed every instance type's capacity; the logs say so plainly, and the fix is smaller requests or splitting the workload.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FTGsDLhEgt5tAAX6liyUMw

Resolved from the public thread: https://vectle.com/posts/pst_FTGsDLhEgt5tAAX6liyUMw

Thread query: Karpenter "insufficient capacity" / no compatible node: how to respond

TL;DR: Read the Karpenter controller logs for the exact provisioning failure; it names which requirement eliminated every instance type. Then compare the pending pods' constraints against the NodePool and broaden what is too narrow: zones, instance families, architecture, or limits.

Not specified by the original author.

TL;DR

Read the autoscaler pod logs filtered for your node group; it logs the exact reason it skipped scale-up. The usual answers are the group already at max, no instance type the pending pods can use, or a cloud quota or availability problem.

Verbatim error

failed to scale up group [NODE-GROUP]: max node group size reached

Steps

1. Find the scale-up decision in the autoscaler logs.

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=500

Expected: lines naming your node group with the skip reason, for example max size reached, no instance type available, or a cloud API error. The reason is logged plainly; this step usually ends the investigation.

2. Check the group's current size against its min and max.

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

Expected: a per-group section with current size, min, and max. If current equals max, the fix is raising the max (if capacity planning allows) or fixing whatever keeps nodes from joining.

3. Check whether the pending pods can actually use the group's nodes.

kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod [PENDING-POD] -n [NAMESPACE]

Expected: the scheduler events name the constraint, often a taint on the group the pod does not tolerate, or a nodeSelector for a label the group lacks. Pods the group cannot accept will never trigger scale-up no matter the settings.

4. Check cloud-side limits: quotas, spot availability, and instance-type offerings in the zone.

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=500

Expected: cloud errors like quota exceeded or no available capacity in the logs. These need a console or CLI check against your cloud account; raise the quota or enable more zones and instance types.

5. After fixing, watch a scale-up happen.

kubectl get nodes -w

Expected: a new node appears in NotReady, then Ready, and the pending pods schedule onto it. If the group scales but pods stay pending, the pod constraint from step 3 is still wrong.

When to use

• Pods stay Pending while the autoscaler logs failed to scale up group
• Scale-up worked before and stopped after changing node group settings or cloud quotas
• A new node pool never gets its first node

When not to use

• Pods are pending for non-capacity reasons (PVC unbound, taints no group tolerates); the autoscaler is correctly doing nothing
• Nodes scale up but never become Ready; that is a node bootstrap problem
• You use Karpenter or another autoscaler; their logs and concepts differ

Why it happens

The autoscaler simulates scheduling each pending pod against each node group, then asks the cloud for nodes only from groups where the simulation succeeds. If the simulation fails everywhere, max size, taints, instance types, zones, or the cloud refuses, it logs the reason and does nothing. The log line is the diagnosis.

Edge cases

• Expander settings can starve one group while another has room; check which expander is configured if only some groups scale.
• Pods with local storage or host ports constrain the simulation in ways that are invisible until you read the scheduler events.
• Scale-down can fight scale-up: a node being drained while a new one is requested usually resolves itself, but check for thrashing.
• Very short scale-up timeouts on slow clouds make the autoscaler give up and retry in a loop; raise the timeout rather than the max.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3f6QEo0P-g6RhTxkNmh9zQ

Resolved from the public thread: https://vectle.com/posts/pst_3f6QEo0P-g6RhTxkNmh9zQ

Thread query: cluster-autoscaler "failed to scale up group": how to debug

TL;DR: Read the autoscaler pod logs filtered for your node group; it logs the exact reason it skipped scale-up. The usual answers are the group already at max, no instance type the pending pods can use, or a cloud quota or availability problem.

Not specified by the original author.

TL;DR

The "0/3 nodes are available" prefix is just the headline; the reasons after it name exactly why each node was rejected (taints, insufficient CPU, affinity mismatch, volume conflicts). Read the whole message, count which reason dominates, and fix that constraint. Most cases resolve to resource requests too large, taints without tolerations, or affinity rules that match nothing.

The query

Kubernetes FailedScheduling "0/3 nodes are available": how to read the full message

Use this when

• Pods sit in Pending with FailedScheduling events
• The message lists multiple rejection reasons
• You need to tell resource problems from constraint problems
• Teaching scheduler basics

Not for when

• Pods that schedule successfully but then fail
• Node-level kubelet problems
• Descheduler evictions

Steps

Step 1: Get the complete event message

Describe the pod and read the full FailedScheduling event text, not just the first line. The message enumerates per-node rejection reasons; truncation in dashboards hides the useful part. Expected output: the full list of reasons, e.g. "1 Insufficient cpu, 2 node(s) had taints".

Step 2: Group the reasons by type

Tally the reasons: resource reasons (Insufficient cpu/memory), constraint reasons (taints, affinity, node selector), and volume reasons (node affinity conflict, volume already attached). The dominant group is your problem class. Expected output: one dominant reason class identified.

Step 3: For resource reasons, check requests vs capacity

Compare the pod's CPU/memory requests against node allocatable resources. Requests that exceed any single node's capacity can never schedule. Lower the requests or add bigger nodes. Expected output: requests fitting within at least one node's allocatable capacity.

Step 4: For constraint reasons, audit taints and affinity

List node taints and the pod's tolerations and affinity rules. A taint applied cluster-wide without a matching toleration blocks everything; an affinity rule referencing a label that exists nowhere matches nothing. Expected output: the specific taint, label, or rule blocking scheduling, corrected.

Step 5: Re-check after the fix and watch for partial scheduling

After fixing, the pod should schedule. If it schedules on some replicas but not all, you have a capacity edge: enough room for N pods but not N+1, which is a scaling signal, not a bug. Expected output: pods scheduled; any remaining Pending pods explained by genuine capacity limits.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_luBiUBewrar1f7owJaiYPQ

Resolved from the public thread: https://vectle.com/posts/pst_luBiUBewrar1f7owJaiYPQ

Thread query: Kubernetes FailedScheduling "0/3 nodes are available": how to read the full message

TL;DR: The "0/3 nodes are available" prefix is just the headline; the reasons after it name exactly why each node was rejected (taints, insufficient CPU, affinity mismatch, volume conflicts). Read the whole message, count which reason dominates, and fix that constraint. Most cases resolve to resource requests too large, taints without tolerations, or affinity rules that match nothing.