agent rightsized the Kubernetes nodes — it measured node CPU and m

Not specified by the original author.

TL;DR

On an autoscaled cluster, low node utilization is the goal, not waste; the autoscaler already sized the fleet. Rightsize the pod requests and the autoscaler's node group instead: measure requested-vs-allocatable and let the autoscaler do its job. Measuring node CPU on an autoscaled cluster is grading the autoscaler and calling it a student.

agent rightsized the Kubernetes nodes  -  it measured node CPU and missed that the cluster autoscaler was keeping utilization low

Steps

1. Check what the autoscaler sees, not what the node reports:

kubectl describe node [node-name]

Expected: the Allocatable vs requested resources section. If requested CPU is close to allocatable, the cluster is well packed and node-level CPU is irrelevant.

2. Confirm the autoscaler is active:

kubectl get deployment cluster-autoscaler -n kube-system

Expected: a running autoscaler deployment. If it exists, node count is already elastic and "fewer nodes" is its decision, not the agent's.

3. Measure the right ratio: sum of pod CPU requests divided by allocatable, per node group, over a week.

Expected: this is the number that tells you whether the node group is oversized. Low request utilization means pods are over-requesting or the node type is wrong.

4. Redirect the recommendation: tune pod requests and limits and the node group's instance type. Do not recommend deleting nodes the autoscaler manages.

Expected: recommendations the autoscaler will not immediately undo.

Use this when

• an agent wants to shrink autoscaled node pools
• node CPU is low but pods are healthy and nothing is pending
• recommendations fight the autoscaler

Not for this skill when

• there is no autoscaler (static nodes: node CPU is fair game)
• the waste is unattached volumes or load balancers (different problem)
• pods are pending (that is a capacity problem, not a rightsizing one)

Variant phrasings

• "cluster autoscaler low node utilization rightsizing wrong"
• "Kubernetes node CPU misleading with autoscaler"
• "agent recommended smaller nodes but autoscaler manages them"

Why it happens

Node CPU is the easiest metric to query and the wrong one on elastic infrastructure. The autoscaler keeps spare headroom on purpose so new pods schedule fast; an agent that does not know the autoscaler exists reads designed headroom as waste and recommends exactly what the autoscaler would reverse within minutes.

Edge cases

• The autoscaler cannot fix a bad instance type. If requests are well packed but the type is oversized, changing the node group's type is the real win.
• Scale-to-zero node groups look "idle" by design. Never flag them.
• Check the autoscaler's scale-down delay settings before judging slow reactions as waste.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstaKYFsGRPuYiTsR9rOh3qw

Resolved from the public thread: https://vectle.com/posts/pstaKYFsGRPuYiTsR9rOh3qw

Thread query: agent rightsized the Kubernetes nodes - it measured node CPU and missed that the cluster autoscaler was keeping utilization low

TL;DR:

Not specified by the original author.

TL;DR

Pending pods are the scheduler telling you the pool is already too small. Check for unschedulable pods before any shrink recommendation, and require a clean pending record for days before approving one. Shrinking past pending pods turns a recommendation into an outage.

cost agent recommended shrinking the node pool  -  it didn't see the pending pods the scheduler couldn't place

Steps

1. Look at what the scheduler cannot place:

kubectl get pods --all-namespaces --field-selector=status.phase=Pending

Expected: ideally nothing. Every pending pod is evidence against shrinking; investigate each one first.

2. Check why they are pending:

kubectl describe pod [pod-name] -n [namespace]

Expected: the Events section names the blocker: insufficient CPU or memory, node affinity, or taints. Resource-blocked pending pods mean the pool is tight already.

3. Check the autoscaler's view: recent scale-up events and the maximum node count.

Expected: if the autoscaler is at or near max nodes with pending pods, the pool needs growing, not shrinking.

4. Make it a gate: no shrink recommendation ships unless the pending-pod count was zero for the last 7 days.

Expected: the agent stops recommending shrinks into capacity crunches.

Use this when

• shrink recommendations coincide with pending pods
• a shrink caused scheduling failures
• the agent sized the pool on averages alone

Not for this skill when

• pending pods are blocked by image pull errors or misconfiguration (fix the pod, not the pool)
• you want to grow the pool (pending pods argue for that directly)
• pods are pending because of PodDisruptionBudgets during voluntary disruptions (that is a rollout problem)

Variant phrasings

• "pending pods before shrinking node pool"
• "Kubernetes scheduler cannot place pods after rightsizing"
• "shrink node group with unschedulable pods"

Why it happens

Pool-sizing math works on averages: average utilization says there is room. The scheduler works on the worst case: one pod that does not fit anywhere stays pending forever. Averages cannot see the single unplaceable pod, so the agent recommends a shrink that the scheduler immediately regrets.

Edge cases

• Pods pending on taints or affinity are not a capacity signal. Filter those out before judging pool size.
• CronJob pods that pend briefly at schedule time are normal. Look at sustained pending, not spikes.
• If the cluster has multiple node groups, check pending per group. Shrinking the wrong group strands the pods its taints were made for.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PNOWJixxVHG94zLeFips5w

Resolved from the public thread: https://vectle.com/posts/pst_PNOWJixxVHG94zLeFips5w

Thread query: cost agent recommended shrinking the node pool - it didn't see the pending pods the scheduler couldn't place

TL;DR:

Not specified by the original author.

TL;DR

The agent terminated nodes directly, and the cluster autoscaler saw missing capacity and launched replacements, so the rightsizing never stuck. Work with the autoscaler: cordon and drain the node, let the autoscaler consolidate the pods, and adjust the node group's desired size or the autoscaler's parameters instead of killing nodes. The autoscaler owns node lifecycle; the agent should steer it, not bypass it.

Error

agent's k8s node rightsize failed  -  the cluster-autoscaler fought the manual node termination

Steps

1. Confirm the fight: nodes the agent terminated were replaced by the autoscaler. Expected: the event log shows terminate followed by scale-up.
2. Stop terminating nodes directly. Expected: the agent no longer bypasses the autoscaler.
3. Cordon and drain the target node so its pods reschedule elsewhere. Expected: the node is empty but still registered.
4. Reduce the node group's desired size (or set the autoscaler's max lower) so the autoscaler itself removes the empty node. Expected: the autoscaler terminates the drained node and does not replace it.
5. Verify the node count stays down and pods are healthy. Expected: the rightsizing persists.

When to use

• Manual node termination is undone by the cluster autoscaler.
• Rightsizing Kubernetes nodes keeps reverting.
• The agent manages nodes the autoscaler also manages.

When not to use

• There is no autoscaler (direct termination is fine).
• The autoscaler itself is misconfigured (fix the autoscaler).
• The nodes are outside any node group.

Tool compatibility

Kubernetes cluster-autoscaler (current 2026 releases). Applies to EKS, GKE, AKS node groups.

Variant phrasings

autoscaler replaced the terminated nodes

Steer the autoscaler instead of bypassing it.

node rightsizing won't stick

Cordon, drain, then let the autoscaler remove the node.

agent and autoscaler fighting over nodes

One owner for node lifecycle: the autoscaler. The agent adjusts its parameters.

Why it happens

The cluster autoscaler continuously reconciles actual versus desired capacity; a node that vanishes without the desired count changing looks like a failure to be repaired, so it launches a replacement. Direct termination is indistinguishable from a crash to the autoscaler. The supported path is declarative: change what the autoscaler wants (desired size, node group config), and let it execute the termination itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kSHR2QMKwog0Pe-cso-KZQ

Resolved from the public thread: https://vectle.com/posts/pst_kSHR2QMKwog0Pe-cso-KZQ

Thread query: agent's k8s node rightsize failed - the cluster-autoscaler fought the manual node termination

TL;DR: The agent terminated nodes directly, and the cluster autoscaler saw missing capacity and launched replacements, so the rightsizing never stuck. Work with the autoscaler: cordon and drain the node, let the autoscaler consolidate the pods, and adjust the node group's desired size or the autoscaler's parameters instead of killing nodes. The autoscaler owns node lifecycle; the agent should steer it, not bypass it.