# Kubernetes pod Failed with reason Evicted: node-pressure eviction

The kubelet killed the pod to relieve node pressure: low memory.available, low nodefs/imagefs disk or inodes, or exhausted PIDs. It sets the pod phase to Failed with reason Evicted, picking victims whose usage exceeds requests first, and leaves the dead pod object in the API. The fix is honest requests, not retrying.

> Confidence: high · Verified: 2026-08-20 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-pod-evicted-node-pressure

## Error signature

```
Status: Failed, Reason: Evicted
```

Codes: Evicted, MemoryPressure, DiskPressure, PIDPressure

## Problem

Pods die with status Evicted, often several at once on the same node, and kubectl logs on the dead pod usually returns nothing because the containers are gone. A Deployment or StatefulSet replaces them, so the workload limps along while the evicted pod objects pile up in kubectl get pods output. The pod that got killed is frequently not the one that caused the pressure — the kubelet ranks victims by how far usage exceeds requests, so a well-behaved pod with no requests set can die for a neighbour's appetite.

## Root cause

- **Node memory pressure hit a pod using more memory than it requested** _(primary)_
  - memory.available on the node fell below the eviction threshold (default hard threshold 100Mi on Linux), and this pod was Burstable or BestEffort with usage above its memory request. The kubelet evicts those first, ordered by priority and then by how far usage exceeds the request — QoS class itself is only a proxy for that ordering.
  - How to tell: kubectl describe pod shows 'The node was low on resource: memory', and the node reported a MemoryPressure condition at eviction time
- **The pod exceeded its own ephemeral-storage limit or an emptyDir sizeLimit** _(primary)_
  - Writable-layer files, logs, and emptyDir volumes count against ephemeral-storage. If a container's usage exceeds its ephemeral-storage limit, or the pod's total exceeds the summed limits, or an emptyDir outgrows its sizeLimit, the kubelet evicts the pod regardless of how much free disk the node still has.
  - How to tell: The pod's status message names the pod itself — 'exceeded its local ephemeral storage limit' or an emptyDir usage message — rather than saying the node was low on a resource
- **Node disk pressure on nodefs or imagefs selected the pod by disk usage** _(common)_
  - nodefs.available (default threshold 10%) or imagefs.available (default 15%) dropped too low. The kubelet first reclaims node-level resources such as unused images; if that is not enough it evicts pods ranked by their disk usage on the starved filesystem — local volumes, logs, and writable layers. QoS-based reasoning does not apply here because ephemeral storage has no QoS classification.
  - How to tell: The message reads 'The node was low on resource: ephemeral-storage' and kubectl describe node shows DiskPressure; df on the node's kubelet and image filesystems confirms which one crossed its threshold
- **Missing or dishonest resource requests put the pod first in line** _(common)_
  - A pod with no memory request is BestEffort, and any usage at all exceeds a zero request — so it sorts to the front of the eviction order on every memory pressure event. Guaranteed pods (requests equal to limits on every container) are never evicted for another pod's consumption, which is why the same neighbours always survive.
  - How to tell: The evicted pods are consistently the ones without requests set, while pods whose requests equal their limits on the same node stay Running through the event
- **PID or inode exhaustion on the node** _(edge)_
  - pid.available or an inodesFree signal crossed its threshold — typically a fork bomb or a workload creating millions of tiny files. PIDs and inodes have no per-pod requests, so the kubelet falls back to ranking victims purely by pod priority.
  - How to tell: kubectl describe node shows PIDPressure, or the eviction message names inodes; disk space and memory on the node look healthy
- **System daemons overran their reservations, squeezing correct pods** _(edge)_
  - If kubelet, journald or other node daemons use more than what system-reserved and kube-reserved set aside, the kubelet can be forced to evict pods that are within their requests — even Guaranteed ones — choosing the lowest priority first. The pods were configured correctly; the node's accounting was not.
  - How to tell: Pods using less than their requests were still evicted; node monitoring shows system-slice processes consuming more than the configured reservations

## Solution

1. Read the eviction message on the dead pod. It names the starved resource and whether the pod breached its own limit or the node ran short — the two need opposite fixes.

```bash
kubectl describe pod <pod-name> -n <namespace> | grep -A4 'Status\|Reason\|Message'
```

   Note: 'The node was low on resource: ...' means node pressure; 'exceeded its local ephemeral storage limit' means the pod broke its own limit.
2. Inspect the node it ran on. Conditions and events show which signal fired (MemoryPressure, DiskPressure, PIDPressure) and whether it is still firing.

```bash
kubectl describe node <node-name> | sed -n '/Conditions:/,/Addresses:/p'
```

3. For memory evictions, set requests near real usage, and make pods you cannot afford to lose Guaranteed by setting requests equal to limits on every container. Guaranteed pods are not evicted for another pod's consumption.

```yaml
resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"   # equal to request => Guaranteed QoS
    cpu: "250m"
```

   Note: Guaranteed also means the container is OOM-killed if it exceeds its own limit — the protection is against neighbours, not against yourself.
4. For ephemeral-storage evictions, bound what the pod may write: set ephemeral-storage requests and limits, cap emptyDir volumes with sizeLimit, and find what is actually writing — logs and cache directories are the usual culprits.

```yaml
containers:
  - name: app
    resources:
      requests:
        ephemeral-storage: "1Gi"
      limits:
        ephemeral-storage: "2Gi"
volumes:
  - name: scratch
    emptyDir:
      sizeLimit: 500Mi
```

5. For node-level disk pressure, reclaim space below the threshold: prune unused images, check that container log rotation is working, and size node disks for the image set they actually carry. The kubelet garbage-collects images itself, but only once pressure already exists.

```bash
df -h /var/lib/kubelet /var/lib/containerd && crictl rmi --prune
```

6. Delete the lingering Evicted pod objects. They stay in the API until removed by hand or by the pod garbage collector, which only runs once terminated pods exceed the kube-controller-manager's terminated-pod-gc-threshold — 12500 by default, so on most clusters they effectively never age out on their own.

```bash
kubectl delete pods --field-selector=status.phase=Failed -n <namespace>
```

   Note: Read their messages first — the evicted pods are the evidence for which signal fired and how often.
7. Protect what matters next time: give critical workloads a PriorityClass (priority breaks ties within an eviction band, and is the only ordering for PID and inode evictions), and do not rely on PodDisruptionBudgets — the kubelet ignores PDBs during node-pressure eviction.

**Verify:** The replacement pods stay Running through the load that previously triggered evictions, kubectl describe node shows MemoryPressure/DiskPressure/PIDPressure all False, and no new pods with reason Evicted appear over a full traffic cycle.

**If that fails:** If honest requests simply do not fit the nodes, the cluster is overcommitted: add capacity or enable an autoscaler. Where evictions fire while dashboards still show free resources, review the kubelet's eviction thresholds and system-reserved/kube-reserved settings — the kubelet acts on its own signals, not on your monitoring.

## Applies to

- Kubernetes: 1.20 and later — Signals and default thresholds are stable; the containerfs signals for split image filesystems are newer (beta since v1.31, gated by KubeletSeparateDiskGC).
- kubelet: 1.20 and later — Hard-threshold evictions use a 0s grace period; PodDisruptionBudgets and terminationGracePeriodSeconds are not respected.
- Platforms: linux/amd64, linux/arm64, windows (memory.available threshold differs, 500Mi default)

## Not applicable to

- OOMKilled with exit code 137, where the kernel killed one container for exceeding its own memory limit — a cgroup kill, not a pod eviction (see container-exit-code-137-oomkilled)
- API-initiated eviction via the Eviction API or kubectl drain, which respects PodDisruptionBudgets and comes from the control plane, not from kubelet pressure signals
- Scheduler preemption, where a pending higher-priority pod displaces a running one at scheduling time — the node was not under resource pressure
- Pods stuck Pending with insufficient cpu/memory, which is a scheduling-time shortfall; an evicted pod ran and was killed
- Taint-based evictions by the node lifecycle controller (node NotReady or unreachable), which are driven by node health, not resource thresholds

## Evidence

1. [Node-pressure Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: That node-pressure eviction sets the pod phase to Failed and terminates it, and that BestEffort/Burstable pods whose usage exceeds requests are evicted first, ordered by priority and then by usage over requests. The same page defines the signals (memory.available, nodefs/imagefs/containerfs, pid.available), the default hard thresholds, the priority-only ordering for inode and PID starvation, and the system-daemon-overrun case.
   > During a node-pressure eviction, the kubelet sets the phase for the selected pods to Failed, and terminates the Pod ... BestEffort or Burstable pods where the usage exceeds requests. These pods are evicted based on their Priority
2. [Pod Quality of Service Classes](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: The practical eviction order by QoS class — BestEffort first, then Burstable, then Guaranteed — that only pods exceeding requests are candidates under resource pressure, and that Guaranteed requires requests equal to limits for every container.
   > When a Node runs out of resources, Kubernetes will first evict BestEffort Pods running on that Node, followed by Burstable and finally Guaranteed Pods.
3. [Ephemeral storage](https://kubernetes.io/docs/concepts/storage/ephemeral-storage/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: That the kubelet evicts a pod whose container writable-layer and log usage exceeds its ephemeral-storage limit, and likewise when the pod total (including emptyDir volumes) exceeds the summed container limits — an eviction the pod triggers on itself, independent of node-wide pressure.
   > For container-level isolation, if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction.
4. [Pod Lifecycle — Garbage collection of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) — The Kubernetes Authors (official-docs), read 2026-08-20
   Supports: That failed pod objects — including evicted ones — remain in the API until explicitly removed, and that the pod garbage collector only cleans terminated pods once their count exceeds terminated-pod-gc-threshold.
   > For failed Pods, the API objects remain in the cluster's API until a human or controller process explicitly removes them.

## Confidence

high — The eviction signals, default thresholds, victim ordering, Guaranteed-pod guarantee, ephemeral-storage eviction rules and the lingering Failed pod object are all quoted from or directly stated on Kubernetes' own documentation pages. The exact status message strings used as discriminators ('The node was low on resource: ...', 'exceeded its local ephemeral storage limit') come from the kubelet's eviction code as observed in practice rather than from a quotable documentation sentence, as does the 12500 default for terminated-pod-gc-threshold (a kube-controller-manager flag default).

## If this is not your failure

Look yours up instead of adapting this one: https://knowbase.sh/search.json?q=<your error text>

To narrow the causes above to the one you have, run the discriminator on each and POST
`{"slug":"kubernetes-pod-evicted-node-pressure","observations":"<what they returned>"}` to https://knowbase.sh/diagnose.json.
It answers with the cause your observations identify and why the others are excluded.

When diagnosis returns an identifiedResolution, run that recipe and its verification criteria,
then POST the completion body it supplies to https://knowbase.sh/outcome.json.
Only a completion response with status=resolved may be presented as resolved; its receipt is agent-observed, not independently verified.

The same workflow is available as MCP tools: https://knowbase.sh/mcp

---

Retrieved from https://knowbase.sh/k/kubernetes-pod-evicted-node-pressure · knowbase · CC-BY-4.0