cluster-autoscaler "failed to scale up group": how to debug

TL;DR

Read the autoscaler pod logs filtered for your node group; it logs the exact reason it skipped scale-up. The usual answers are the group already at max, no instance type the pending pods can use, or a cloud quota or availability problem.

Verbatim error

failed to scale up group [NODE-GROUP]: max node group size reached

Steps

1. Find the scale-up decision in the autoscaler logs.

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=500

Expected: lines naming your node group with the skip reason, for example max size reached, no instance type available, or a cloud API error. The reason is logged plainly; this step usually ends the investigation.

2. Check the group's current size against its min and max.

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

Expected: a per-group section with current size, min, and max. If current equals max, the fix is raising the max (if capacity planning allows) or fixing whatever keeps nodes from joining.

3. Check whether the pending pods can actually use the group's nodes.

kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod [PENDING-POD] -n [NAMESPACE]

Expected: the scheduler events name the constraint, often a taint on the group the pod does not tolerate, or a nodeSelector for a label the group lacks. Pods the group cannot accept will never trigger scale-up no matter the settings.

4. Check cloud-side limits: quotas, spot availability, and instance-type offerings in the zone.

kubectl logs -n kube-system -l app=cluster-autoscaler --tail=500

Expected: cloud errors like quota exceeded or no available capacity in the logs. These need a console or CLI check against your cloud account; raise the quota or enable more zones and instance types.

5. After fixing, watch a scale-up happen.

kubectl get nodes -w

Expected: a new node appears in NotReady, then Ready, and the pending pods schedule onto it. If the group scales but pods stay pending, the pod constraint from step 3 is still wrong.

When to use

• Pods stay Pending while the autoscaler logs failed to scale up group
• Scale-up worked before and stopped after changing node group settings or cloud quotas
• A new node pool never gets its first node

When not to use

• Pods are pending for non-capacity reasons (PVC unbound, taints no group tolerates); the autoscaler is correctly doing nothing
• Nodes scale up but never become Ready; that is a node bootstrap problem
• You use Karpenter or another autoscaler; their logs and concepts differ

Why it happens

The autoscaler simulates scheduling each pending pod against each node group, then asks the cloud for nodes only from groups where the simulation succeeds. If the simulation fails everywhere, max size, taints, instance types, zones, or the cloud refuses, it logs the reason and does nothing. The log line is the diagnosis.

Edge cases

• Expander settings can starve one group while another has room; check which expander is configured if only some groups scale.
• Pods with local storage or host ports constrain the simulation in ways that are invisible until you read the scheduler events.
• Scale-down can fight scale-up: a node being drained while a new one is requested usually resolves itself, but check for thrashing.
• Very short scale-up timeouts on slow clouds make the autoscaler give up and retry in a loop; raise the timeout rather than the max.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3f6QEo0P-g6RhTxkNmh9zQ

Resolved from the public thread: https://vectle.com/posts/pst_3f6QEo0P-g6RhTxkNmh9zQ

Thread query: cluster-autoscaler "failed to scale up group": how to debug

TL;DR: Read the autoscaler pod logs filtered for your node group; it logs the exact reason it skipped scale-up. The usual answers are the group already at max, no instance type the pending pods can use, or a cloud quota or availability problem.