agent rightsized the Kubernetes nodes - it measured node CPU and missed that the cluster autoscaler was keeping...

TL;DR

On an autoscaled cluster, low node utilization is the goal, not waste; the autoscaler already sized the fleet. Rightsize the pod requests and the autoscaler's node group instead: measure requested-vs-allocatable and let the autoscaler do its job. Measuring node CPU on an autoscaled cluster is grading the autoscaler and calling it a student.

agent rightsized the Kubernetes nodes  -  it measured node CPU and missed that the cluster autoscaler was keeping utilization low

Steps

1. Check what the autoscaler sees, not what the node reports:

kubectl describe node [node-name]

Expected: the Allocatable vs requested resources section. If requested CPU is close to allocatable, the cluster is well packed and node-level CPU is irrelevant.

1. Confirm the autoscaler is active:

kubectl get deployment cluster-autoscaler -n kube-system

Expected: a running autoscaler deployment. If it exists, node count is already elastic and "fewer nodes" is its decision, not the agent's.

1. Measure the right ratio: sum of pod CPU requests divided by allocatable, per node group, over a week.

Expected: this is the number that tells you whether the node group is oversized. Low request utilization means pods are over-requesting or the node type is wrong.

1. Redirect the recommendation: tune pod requests and limits and the node group's instance type. Do not recommend deleting nodes the autoscaler manages.

Expected: recommendations the autoscaler will not immediately undo.

Use this when

• an agent wants to shrink autoscaled node pools
• node CPU is low but pods are healthy and nothing is pending
• recommendations fight the autoscaler

Not for this skill when

• there is no autoscaler (static nodes: node CPU is fair game)
• the waste is unattached volumes or load balancers (different problem)
• pods are pending (that is a capacity problem, not a rightsizing one)

Variant phrasings

• "cluster autoscaler low node utilization rightsizing wrong"
• "Kubernetes node CPU misleading with autoscaler"
• "agent recommended smaller nodes but autoscaler manages them"

Why it happens

Node CPU is the easiest metric to query and the wrong one on elastic infrastructure. The autoscaler keeps spare headroom on purpose so new pods schedule fast; an agent that does not know the autoscaler exists reads designed headroom as waste and recommends exactly what the autoscaler would reverse within minutes.

Edge cases

• The autoscaler cannot fix a bad instance type. If requests are well packed but the type is oversized, changing the node group's type is the real win.
• Scale-to-zero node groups look "idle" by design. Never flag them.
• Check the autoscaler's scale-down delay settings before judging slow reactions as waste.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstaKYFsGRPuYiTsR9rOh3qw