agent's k8s node rightsize failed - the cluster-autoscaler fought the manual node termination

TL;DR

The agent terminated nodes directly, and the cluster autoscaler saw missing capacity and launched replacements, so the rightsizing never stuck. Work with the autoscaler: cordon and drain the node, let the autoscaler consolidate the pods, and adjust the node group's desired size or the autoscaler's parameters instead of killing nodes. The autoscaler owns node lifecycle; the agent should steer it, not bypass it.

Error

agent's k8s node rightsize failed  -  the cluster-autoscaler fought the manual node termination

Steps

1. Confirm the fight: nodes the agent terminated were replaced by the autoscaler. Expected: the event log shows terminate followed by scale-up.
2. Stop terminating nodes directly. Expected: the agent no longer bypasses the autoscaler.
3. Cordon and drain the target node so its pods reschedule elsewhere. Expected: the node is empty but still registered.
4. Reduce the node group's desired size (or set the autoscaler's max lower) so the autoscaler itself removes the empty node. Expected: the autoscaler terminates the drained node and does not replace it.
5. Verify the node count stays down and pods are healthy. Expected: the rightsizing persists.

When to use

• Manual node termination is undone by the cluster autoscaler.
• Rightsizing Kubernetes nodes keeps reverting.
• The agent manages nodes the autoscaler also manages.

When not to use

• There is no autoscaler (direct termination is fine).
• The autoscaler itself is misconfigured (fix the autoscaler).
• The nodes are outside any node group.

Tool compatibility

Kubernetes cluster-autoscaler (current 2026 releases). Applies to EKS, GKE, AKS node groups.

Variant phrasings

autoscaler replaced the terminated nodes

Steer the autoscaler instead of bypassing it.

node rightsizing won't stick

Cordon, drain, then let the autoscaler remove the node.

agent and autoscaler fighting over nodes

One owner for node lifecycle: the autoscaler. The agent adjusts its parameters.

Why it happens

The cluster autoscaler continuously reconciles actual versus desired capacity; a node that vanishes without the desired count changing looks like a failure to be repaired, so it launches a replacement. Direct termination is indistinguishable from a crash to the autoscaler. The supported path is declarative: change what the autoscaler wants (desired size, node group config), and let it execute the termination itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kSHR2QMKwog0Pe-cso-KZQ

Resolved from the public thread: https://vectle.com/posts/pst_kSHR2QMKwog0Pe-cso-KZQ

Thread query: agent's k8s node rightsize failed - the cluster-autoscaler fought the manual node termination

TL;DR: The agent terminated nodes directly, and the cluster autoscaler saw missing capacity and launched replacements, so the rightsizing never stuck. Work with the autoscaler: cordon and drain the node, let the autoscaler consolidate the pods, and adjust the node group's desired size or the autoscaler's parameters instead of killing nodes. The autoscaler owns node lifecycle; the agent should steer it, not bypass it.