# Kubernetes pod stuck in Terminating and never deleted Terminating means deletionTimestamp is set and the API server is waiting — for finalizers to clear, or for the kubelet to confirm the containers stopped. When either never happens the pod stays forever. Check metadata.finalizers and node health before reaching for --force. > Confidence: high · Verified: 2026-08-11 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-pod-stuck-terminating ## Error signature ``` Terminating ``` Codes: Terminating ## Problem kubectl delete returned immediately, but minutes or hours later the pod is still listed with STATUS Terminating, and deleting it again changes nothing. The delete was accepted, not performed: the API server set metadata.deletionTimestamp and is now waiting — for every finalizer on the pod to be removed by its controller, and for the kubelet to confirm the containers actually stopped. The pod stays visible until both happen, and either one can be blocked indefinitely. ## Root cause - **A finalizer on the pod is never removed** _(primary)_ - The controller that owns the finalizer key — a service mesh, a backup agent, a custom operator — crashed, was uninstalled, or lost the RBAC permission to patch the pod. Once deletion is requested nothing new can be added to the finalizers list, but every existing key blocks deletion until its controller clears it, and an absent controller clears nothing. - How to tell: kubectl get pod -o jsonpath='{.metadata.finalizers}' prints a non-empty list, and the component named by the key is unhealthy or no longer installed - **The node's kubelet is unreachable, so deletion is never confirmed** _(primary)_ - Graceful deletion completes only when the kubelet on the pod's node kills the containers and reports back to the API server. A powered-off, partitioned, or non-gracefully shut down node never reports, so the pod object waits on it forever — Kubernetes will not guess that the workload is dead. - How to tell: kubectl get node for the pod's .spec.nodeName shows NotReady or Unknown, and every pod on that node is stuck the same way - **A volume unmount or detach hangs** _(common)_ - The containers are gone but the kubelet cannot finish cleanup because a volume will not unmount — a dead NFS server, a crashed CSI node plugin, or a mount held busy by a leaked process. Kubernetes only force-detaches volumes after a deletion has failed for six minutes, and only if the node is unhealthy at that moment, so a healthy node with a wedged mount waits indefinitely. - How to tell: The kubelet log on the node repeats unmount or UnmountVolume errors naming the pod's volumes, or the CSI node plugin pod on that node is not Running - **A container process survives SIGKILL** _(common)_ - A process in uninterruptible sleep — D state, typically blocked on I/O against a dead network mount or failing disk — cannot be killed by any signal. The runtime can never report the container as exited, so the kubelet never confirms termination, however many grace periods expire. - How to tell: On the node, ps shows the container's main process in state D, and crictl ps still lists the container long after the grace period expired - **It is not stuck, only slow — a long grace period or preStop hook** _(edge)_ - A terminationGracePeriodSeconds of an hour, or a preStop hook that drains connections slowly, keeps the pod legitimately in Terminating for that long. Nothing is wrong; the deletion completes when the period expires. - How to tell: .spec.terminationGracePeriodSeconds is large and the time elapsed since deletionTimestamp is still inside it — waiting resolves it without any action ## Solution 1. Establish which wait you are in before touching anything. One command shows the deletion timestamp, the finalizers, the node, and the grace period — between them they discriminate every cause above. ```bash kubectl get pod -n -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}{.spec.nodeName}{"\n"}{.spec.terminationGracePeriodSeconds}{"\n"}' ``` Note: Then check the node it names: kubectl get node . A NotReady node changes the whole diagnosis. 2. If finalizers are set, fix the controller that owns each key rather than deleting the key. Restart the crashed operator, reinstall the component that was removed, or restore its RBAC — the finalizer then clears itself and the cleanup it guarded actually runs. 3. Only when the owning controller is gone for good, patch the finalizers away — accepting that whatever cleanup the finalizer guaranteed will not happen. This is the documented last step in Kubernetes' own force-delete task, not a first move. ```bash kubectl patch pod -n -p '{"metadata":{"finalizers":null}}' ``` 4. If the node is NotReady, prefer healing the node over forcing the pod. When a network partition resolves, the kubelet completes the deletion on its own. If the node is confirmed dead — powered off, not mid-restart — taint it out-of-service: its pods are then force-deleted and their volumes detached immediately, which is what lets StatefulSet replacements start elsewhere. ```bash # only after confirming the node is shut down, not restarting kubectl taint nodes node.kubernetes.io/out-of-service=nodeshutdown:NoExecute ``` Note: You must remove the taint yourself once the workloads have moved and the node has recovered — nothing removes it automatically. 5. If volumes are the blocker on a healthy node, read the kubelet log and check the CSI node plugin. Restarting a crashed plugin pod usually lets the pending unmount complete; a dead NFS server has to come back or be force-unmounted on the node itself. ```bash ssh 'journalctl -u kubelet --since "15 min ago" | grep -iE "unmount|volume|orphan"' ``` 6. Force delete only as the last resort, and treat StatefulSet pods as a special case. The command removes the pod name from the API immediately, without the kubelet confirming anything — the process may still be running. For a StatefulSet that frees the identity for a replacement while the original may still hold it, so you are asserting the old pod will never talk to its peers again. For a Deployment pod the blast radius is smaller, but the workload may still be running invisibly on the node. ```bash kubectl delete pod -n --grace-period=0 --force ``` Note: Verify the node is genuinely down or the process genuinely dead before force-deleting anything that holds a lock, a lease, or a StatefulSet identity. **Verify:** kubectl get pod -n returns NotFound and the replacement pod reaches Running. For StatefulSet pods, additionally confirm the old node was actually down before the replacement started; for volume-backed pods, confirm the PVC attached cleanly on the new node instead of erroring on a stale attachment. **If that fails:** Where pods from one operator get stuck on its finalizer routinely, treat that as a bug in the operator rather than a per-pod chore — upgrade or remove it, or stop it adding finalizers to pods it cannot reliably clean up. Patching each pod by hand is a treadmill, not a fix. ## Applies to - Kubernetes: 1.20 and later — Terminating is a display state kubectl derives from a set metadata.deletionTimestamp, not a Pod phase. - kubelet: 1.28 and later for the out-of-service taint — Non-graceful node shutdown handling (the out-of-service taint) is stable and enabled by default from v1.28. - Runtimes: containerd, CRI-O - Platforms: linux/amd64, linux/arm64 ## Not applicable to - Evicted pods, which are already terminal and merely left behind as records — nothing is waiting, and deleting them completes instantly - CrashLoopBackOff, where the pod is alive and restarting and nobody has requested deletion - Pods stuck in Pending, a scheduling problem in which the pod never started, let alone stopped - A namespace stuck in Terminating, the same finalizer mechanism on a Namespace object — usually an unreachable aggregated API service — with a different diagnosis and fix ## Evidence 1. [Finalizers](https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That deletion sets metadata.deletionTimestamp and the object then waits in a terminating state until controllers remove every finalizer, that an emptied finalizers list is what completes deletion, and that finalizers can be removed but not added once deletion is requested. > The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers. 2. [Force Delete StatefulSet Pods](https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That force deletion skips kubelet confirmation and frees the pod name immediately, that for StatefulSets this can duplicate a still-running pod and violate at-most-one semantics, and that patching finalizers to null is the documented last step when the pod is still stuck afterwards. > Force deletions do not wait for confirmation from the kubelet that the Pod has been terminated ... this can lead to the duplication of a still-running Pod 3. [Pod Lifecycle — Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: The graceful termination flow — 30-second default grace period, TERM then KILL, kubelet removing the pod object after confirming shutdown — and the caution that --grace-period=0 --force deletes the API object without confirmation while the workload may keep running. > Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. 4. [Node Shutdowns — Non-graceful node shutdown handling](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That pods on a node that shut down non-gracefully stay in terminating status forever if the node never returns, that the node.kubernetes.io/out-of-service taint force-deletes them and detaches their volumes immediately, and that automatic force-detach otherwise requires six minutes plus an unhealthy node. > these pods will be stuck in terminating status on the shutdown node forever ... In any situation where a pod deletion has not succeeded for 6 minutes, kubernetes will force detach volumes being unmounted if the node is unhealthy 5. [Garbage Collection](https://kubernetes.io/docs/concepts/architecture/garbage-collection/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That finalizers are the mechanism by which garbage collection defers deletion until cleanup completes, which is why removing one by hand skips that cleanup. > If a finalizer exists, it ensures that objects are not deleted until all necessary clean-up tasks are completed. ## Confidence high — The finalizer mechanism, force-deletion semantics and their StatefulSet risk, the stuck-forever behaviour on a dead node, and the out-of-service taint are all quoted from Kubernetes' own documentation. The SIGKILL-immune D-state process cause and the CSI-plugin diagnostics rest on kernel semantics and operational practice rather than on sentences these pages contain, and the discriminators are observable checks the author composed, not sourced claims. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"kubernetes-pod-stuck-terminating","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. When diagnosis returns an identifiedResolution, run that recipe and its verification criteria, then POST the completion body it supplies to https://knowbase.sh/outcome.json. Only a completion response with status=resolved may be presented as resolved; its receipt is agent-observed, not independently verified. The same workflow is available as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/kubernetes-pod-stuck-terminating · knowbase · CC-BY-4.0