how to debug a pod stuck in Terminating in Kubernetes

tl;dr

never start with force delete. first check whether the pod has a finalizer holding it, then check whether its node is NotReady. only force-delete when you know the pod is not really running (usually a dead node), because forcing a live pod can orphan running containers.

kubectl get pod [pod-name] -n [namespace]
# NAME        READY   STATUS        RESTARTS   AGE
# [pod-name]  0/1     Terminating   0          47m

47 minutes in Terminating after kubectl delete is the symptom. the rest of this skill walks the three usual causes: a container that ignores SIGTERM, a finalizer waiting on cleanup, or a node that can no longer talk to the kubelet.

steps

1. confirm it is actually stuck. run kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.metadata.deletionTimestamp}'. expected output: an ISO timestamp minutes or hours in the past. if there is no timestamp, the pod was not deleted at all and something else is wrong.

1. look for finalizers. run kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.metadata.finalizers}'. expected output: either empty, or a list like ["kubernetes.io/pv-protection"]. a non-empty list means a controller is waiting on cleanup (usually a volume detach or a custom controller). fix the cleanup, not the pod: kubectl describe pv [pv-name] and kubectl get events -n [namespace] --sort-by=.lastTimestamp usually show what the controller is stuck on.

1. check the node. run kubectl describe pod [pod-name] -n [namespace] | grep Node: to get the node name, then kubectl get node [node-name]. expected output for the stuck case: NotReady or Unknown under STATUS. a dead or unreachable node is the most common reason pods hang in Terminating: the kubelet can no longer confirm the containers stopped, so the API server cannot clear the pod. if the node is Ready, skip to step 5.

1. recover the node, or remove it. if the node will come back, wait for it. if the node is permanently gone (terminated VM, dead hardware), run kubectl delete node [node-name]. after the node is removed the pods on it are released and workloads reschedule. on managed clusters check the node autoscaler first so it does not replace a node you just deleted.

1. rule out a slow container shutdown. if the node is Ready and there are no finalizers, a preStop hook or a process ignoring SIGTERM is the cause. check kubectl get pod [pod-name] -n [namespace] -o jsonpath='{.spec.containers[*].lifecycle}' for a preStop hook, and .spec.terminationGracePeriodSeconds for the timeout the kubelet waits (default 30s) before SIGKILL. fix the container to exit on SIGTERM; do not raise the grace period to hide the problem.

1. last resort: force delete. only when you are sure the pod is not really running. run kubectl delete pod [pod-name] -n [namespace] --grace-period=0 --force. expected output: pod "[pod-name]" force deleted. this tells the API server to drop the pod record without waiting for the kubelet. the warning from kubectl is real: if the pod is alive on a healthy node, you will orphan running containers.

when to use this skill

• kubectl get pods shows a pod in Terminating long after delete
• delete returned success but the pod never disappears
• you need to decide between waiting, fixing a finalizer, removing a node, or force delete

when not to use this skill

• pod is in CrashLoopBackOff, ImagePullBackOff, Pending, or Error: different failure, different fix
• the pod restarts on its own after deletion: that is the controller (Deployment, StatefulSet) doing its job. delete the workload or scale it to zero, not the pod
• you want to stop a deployment quickly in an emergency: use kubectl rollout undo or scale down, not pod deletion

variant phrasings

pod stuck in Terminating forever

same cause set as above. agents and humans phrase this as "hangs", "forever", or "won't terminate"; the debugging order (finalizers, node, shutdown behavior) does not change.

kubectl delete pod hangs or times out

if the kubectl delete command itself hangs rather than returning, the API server is slow to process deletion or your kubeconfig cannot reach it. check kubectl cluster-info first, then continue with the steps above.

cannot delete pod, deletionTimestamp set but pod remains

this is the API-server view of the same problem. a set deletionTimestamp plus a lingering pod record always means the kubelet has not confirmed container stop or a finalizer is outstanding; steps 2 and 3 are the whole story.

root cause

pod deletion is a handshake, not an event. the API server marks the pod with a deletionTimestamp and sends SIGTERM; the kubelet on the node must report the containers stopped, and every finalizer must be removed, before the API server deletes the pod record. any break in the chain (dead node, stuck controller, container ignoring SIGTERM) leaves the pod in Terminating indefinitely.

edge cases

• StatefulSet pods: force-deleting one triggers the controller to recreate it. that is expected, not a bug. delete the StatefulSet or scale it down if you want it gone.
• PV protection finalizer: the pod is waiting on a volume detach. detaching can take minutes on a healthy cluster; on a dead node you may need to force-detach the volume in the cloud console instead.
• node autoscaler loops: deleting a NotReady node on a managed cluster can cause the autoscaler to spin up a replacement immediately. cordon the node first if you only want the pods evicted.
• namespace stuck in Terminating: different problem. check for remaining resources and finalizers on the namespace itself, not the pod.