# Kubernetes node NotReady: pods stop scheduling and get evicted

A node goes NotReady when its kubelet stops posting a healthy status: the kubelet died, the API server is unreachable, the container runtime broke, or the kubelet's client certificate expired. The node is then tainted NoExecute, and pods without a longer toleration are evicted about five minutes later.

> Confidence: high · Verified: 2026-08-20 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-node-notready

## Error signature

```
NotReady
```

Codes: NotReady, NodeNotReady, NodeStatusUnknown, node.kubernetes.io/not-ready, node.kubernetes.io/unreachable

## Problem

kubectl get nodes shows a node as NotReady. The scheduler stops placing new pods on it, and roughly five minutes later the pods it was running begin terminating and reappearing elsewhere — or, if the node is unreachable, hang in Terminating because nothing can confirm they stopped. The STATUS column says nothing about why. The answer is in the node's Ready condition, and the first split is whether it reads False — the kubelet is alive and reporting a problem — or Unknown — the kubelet's heartbeats stopped arriving at all.

## Root cause

- **The kubelet on the node stopped or is crash-looping** _(primary)_
  - The kubelet is the source of both heartbeats — Node status updates and Lease renewals in kube-node-lease. When the process dies (OOM-killed, crashed on a bad flag or config after an upgrade, disabled by mistake), the heartbeats stop and the node controller flips Ready to Unknown after node-monitor-grace-period, 50 seconds by default.
  - How to tell: kubectl describe node shows Ready Unknown with 'Kubelet stopped posting node status', and on the node systemctl status kubelet reports inactive, failed or restarting
- **The node cannot reach the API server** _(primary)_
  - The kubelet is healthy but its heartbeats never arrive — a network partition, a changed firewall or security-group rule, a broken load balancer in front of the control plane, or a rebooted VM that came back on a different network. The node controller sets Ready to Unknown and the node gets the node.kubernetes.io/unreachable taint rather than not-ready.
  - How to tell: The kubelet journal shows repeated 'Failed to update lease', connection refused or i/o timeout against the API server address, and curl -k https://<control-plane-endpoint>:6443/healthz fails from the node while succeeding from elsewhere
- **The container runtime is down or its CRI socket is broken** _(common)_
  - The kubelet keeps posting status but reports itself unhealthy, so Ready reads False rather than Unknown and the message names the runtime. containerd or CRI-O crashed, an upgrade moved the socket path, or a full disk stopped the runtime from writing.
  - How to tell: kubectl describe node shows Ready False with a message naming the container runtime, and crictl info run on the node errors instead of printing runtime status
- **The kubelet's client certificate expired, so the API server rejects it** _(common)_
  - kubeadm-issued client certificates expire after one year. Rotation normally renews them, but a node that was powered off across the renewal window, or a cluster never upgraded, misses it — and then every status update is rejected. A whole cluster's nodes flipping NotReady together on the anniversary of the install is this cause until proven otherwise.
  - How to tell: The kubelet journal or kube-apiserver logs show 'x509: certificate has expired or is not yet valid', and kubeadm certs check-expiration lists expired certificates
- **The CNI network plugin is missing or not initialized on the node** _(common)_
  - A freshly joined node stays NotReady until its CNI plugin is installed and running; an existing node regresses when the CNI daemonset is deleted or its config under /etc/cni/net.d is wiped. The runtime itself is fine — it reports NetworkReady false, and the kubelet relays that as Ready False.
  - How to tell: kubectl describe node shows 'container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady', while crictl info succeeds and /etc/cni/net.d on the node is empty or missing
- **The node itself is starved, so the kubelet cannot heartbeat in time** _(edge)_
  - Memory pressure severe enough that the OOM killer takes the kubelet or the runtime, a completely full disk, or CPU saturation that delays lease renewal past the grace period. The signature is flapping — the node oscillates between Ready and NotReady as the kubelet limps along.
  - How to tell: The node flaps between Ready and NotReady, and on the node dmesg shows oom-killer activity or df reports a full filesystem

## Solution

1. Read the node's conditions and taints. False and Unknown are different diagnoses: False means the kubelet is alive and its message names the problem; Unknown means heartbeats stopped, so the problem is the kubelet process, the node, or the network path to the API server.

```bash
kubectl describe node <node-name>
```

   Note: The Taints section shows which of node.kubernetes.io/not-ready (Ready False) or node.kubernetes.io/unreachable (Ready Unknown) the control plane applied.
2. SSH to the node and check the kubelet process before anything else — a dead kubelet explains every Unknown, and its last log lines say why it stopped.

```bash
systemctl status kubelet
journalctl -u kubelet --since "-30 min" --no-pager | tail -50
```

   Note: Look for a crash loop on startup (bad flag or config after an upgrade), x509 errors (certificate), or connection errors (network, API server).
3. If Ready is False and the message names the container runtime, check the runtime behind the CRI socket rather than the kubelet.

```bash
systemctl status containerd   # or crio
crictl info
```

   Note: crictl info failing means the runtime is down. crictl succeeding but showing NetworkReady false means the runtime is fine and the CNI plugin is the problem — reinstall or repair it instead of restarting the runtime.
4. If Ready is Unknown but the kubelet is running, test the path to the API server from the node itself — a partition looks identical to a dead kubelet from the control plane's side.

```bash
curl -sk https://<control-plane-endpoint>:6443/healthz
```

   Note: Expect ok. A timeout or refusal from the node while the same request works from elsewhere is a firewall, security-group or routing change, not a Kubernetes problem.
5. Check certificate expiry on kubeadm clusters — client certificates expire after one year, and an expired kubelet client certificate silently turns every heartbeat into a rejected request.

```bash
kubeadm certs check-expiration
openssl x509 -enddate -noout -in /var/lib/kubelet/pki/kubelet-client-current.pem
```

6. If the kubelet client certificate is expired, follow the documented repair: remove the stale kubeconfig and client certs from the failed node, mint a new kubelet.conf from a control-plane node holding the CA key, copy it back and restart the kubelet.

```bash
# on the failed node — back up and remove the stale credentials
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak
mv /var/lib/kubelet/pki/kubelet-client* /tmp/

# on a control-plane node that has /etc/kubernetes/pki/ca.key
kubeadm kubeconfig user --org system:nodes \
  --client-name "system:node:$NODE" > kubelet.conf

# copy kubelet.conf to /etc/kubernetes/kubelet.conf on the failed node, then
systemctl restart kubelet
```

   Note: Wait for /var/lib/kubelet/pki/kubelet-client-current.pem to be recreated, then point kubelet.conf back at the rotated certificate so future rotation works again.
7. For workloads that must ride out short node blips, set an explicit toleration — otherwise the default injected tolerationSeconds of 300 evicts them five minutes after the taint lands.

```yaml
tolerations:
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 6000
```

   Note: DaemonSet pods already tolerate both taints with no time limit and are never evicted for them.
8. Fix the underlying cause, restart the kubelet, and watch the node return. The not-ready and unreachable taints are removed automatically when the Ready condition recovers — do not remove them by hand.

```bash
kubectl get nodes -w
```


**Verify:** kubectl get nodes reports the node Ready; kubectl describe node no longer lists node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taints; evicted workloads are Running again and pods on the node stop terminating.

**If that fails:** If the node cannot be recovered — hardware gone, VM deleted, certificate CA lost — drain what the API still believes is there and delete the Node object so controllers reschedule everything: kubectl drain <node> --ignore-daemonsets --force followed by kubectl delete node <node>. Pods stuck Terminating on an unreachable node are released when the Node object goes away.

## Applies to

- Kubernetes: 1.18 and later — Taint-based eviction for node conditions is stable since v1.18; after 1.29 it runs in a separate taint-eviction-controller rather than the node controller.
- kubelet: 1.18 and later — Posts Ready via Node status updates and Lease renewals; the node controller marks it Unknown after node-monitor-grace-period, 50 seconds by default.
- kubeadm: 1.15 and later — Client certificates expire after one year unless renewed by upgrade or kubeadm certs renew.
- Runtimes: containerd, CRI-O
- Platforms: linux/amd64, linux/arm64

## Not applicable to

- Pods Evicted by node-pressure eviction while the node itself stays Ready — that is the kubelet reclaiming memory, disk or PIDs on a healthy node (kubernetes-pod-evicted-node-pressure), not a NotReady node
- Nodes showing SchedulingDisabled after kubectl cordon or drain — an intentional administrative state; the node is still Ready and no taint-based eviction runs
- A single pod stuck Pending on a cluster of Ready nodes, which is a scheduling or resource problem rather than node health
- A Node object deleted by the cloud provider or cluster autoscaler, where the node disappears from kubectl get nodes instead of going NotReady

## Evidence

1. [Node Status — Conditions](https://kubernetes.io/docs/reference/node/node-status/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: The meaning of Ready False versus Unknown, and that the node controller marks a silent node Unknown after node-monitor-grace-period, 50 seconds by default.
   > False if the node is not healthy and is not accepting pods, and Unknown if the node controller has not heard from the node in the last node-monitor-grace-period (default is 50 seconds)
2. [Nodes — Node controller](https://kubernetes.io/docs/concepts/architecture/nodes/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: That the node controller is what sets Ready to Unknown when a node becomes unreachable, and that it then triggers API-initiated eviction of the node's pods.
   > In the case that a node becomes unreachable, updating the Ready condition in the Node's .status field. In this case the node controller sets the Ready condition to Unknown
3. [Taints and Tolerations — Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: That node.kubernetes.io/not-ready and node.kubernetes.io/unreachable are the condition taints, that every pod gets an automatic toleration of 300 seconds for them, and therefore that eviction begins about five minutes after the taint lands.
   > Kubernetes automatically adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300 ... Pods remain bound to Nodes for 5 minutes after one of these problems is detected.
4. [Certificate Management with kubeadm](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: That kubeadm client certificates expire after one year, and that kubeadm certs check-expiration is the command that shows expiry per certificate.
   > Client certificates generated by kubeadm expire after 1 year.
5. [Troubleshooting kubeadm — Kubelet client certificate rotation fails](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: The x509 error string used as the discriminator for the expired-certificate cause, and the documented repair sequence of deleting the stale kubelet.conf and client certificates, minting a new kubeconfig and restarting the kubelet.
   > If this rotation process fails you might see errors such as x509: certificate has expired or is not yet valid in kube-apiserver logs.

## Confidence

high — The condition semantics, the 50-second grace period, the two condition taints, the 300-second automatic toleration and the one-year certificate lifetime are all quoted from kubernetes.io documentation. The False-versus-Unknown diagnostic split follows directly from those documented mechanisms, but the specific journal and log strings used as discriminators for the runtime, network and resource causes — 'Failed to update lease', crictl failure modes, Ready/NotReady flapping under OOM — are operational practice rather than sentences quotable from a single page.

## If this is not your failure

Look yours up instead of adapting this one: https://knowbase.sh/search.json?q=<your error text>

To narrow the causes above to the one you have, run the discriminator on each and POST
`{"slug":"kubernetes-node-notready","observations":"<what they returned>"}` to https://knowbase.sh/diagnose.json.
It answers with the cause your observations identify and why the others are excluded.

When diagnosis returns an identifiedResolution, run that recipe and its verification criteria,
then POST the completion body it supplies to https://knowbase.sh/outcome.json.
Only a completion response with status=resolved may be presented as resolved; its receipt is agent-observed, not independently verified.

The same workflow is available as MCP tools: https://knowbase.sh/mcp

---

Retrieved from https://knowbase.sh/k/kubernetes-node-notready · knowbase · CC-BY-4.0