# Kubernetes Job failed: reached the specified backoff limit The Job controller counted pod failures up to .spec.backoffLimit — 6 by default — and marked the Job permanently Failed with reason BackoffLimitExceeded. The condition is pure accounting: the actual reason lives in the logs of the failed pods, which are kept around precisely so you can read them. > Confidence: high · Verified: 2026-08-11 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-job-backofflimitexceeded ## Error signature ``` Job has reached the specified backoff limit ``` Codes: BackoffLimitExceeded, JobReasonBackoffLimitExceeded ## Problem A Job stops creating pods and kubectl describe job shows a Warning event with reason BackoffLimitExceeded and a Failed condition carrying the message 'Job has reached the specified backoff limit'. The message says only that the failure budget ran out, never why the pods failed. Because retries are spaced with an exponential back-off delay, the Job can look merely slow for many minutes before it dies — with the default limit of 6 the built-in example takes at least nine minutes to fail. Once the condition is set the failure is permanent: nothing restarts a Failed Job, so waiting accomplishes nothing. ## Root cause - **The workload fails deterministically, so every retry fails identically** _(primary)_ - A code bug, wrong arguments, a missing input file or an unreachable dependency makes the container exit non-zero on every attempt. Retrying was never going to help; the backoff limit just decides how long the futile retries continue — roughly nine minutes at the default of 6, thanks to the exponential delay. - How to tell: Every retained failed pod shows the same error and the same exit code — kubectl logs on two different pods of the Job print the same failure - **The container is killed for exceeding resources, counting as a failure each time** _(common)_ - An OOM kill (exit code 137) or an eviction for disk pressure terminates the pod, the Job controller counts it and recreates it, and the replacement dies the same way until the budget is spent. The Job accounting cannot tell an OOM from a bug. - How to tell: kubectl describe pod on a failed pod shows lastState terminated with reason OOMKilled or exitCode 137, not an application error in the logs - **Transient disruptions are counted because no pod failure policy ignores them** _(common)_ - Node drains, preemption and taint-based eviction fail the pod through no fault of the workload. By default each such failure is counted toward .spec.backoffLimit, so cluster maintenance can exhaust a small budget while the program itself never misbehaved. - How to tell: Failed pods carry the DisruptionTarget condition, and the failures cluster around node upgrades or autoscaler scale-downs rather than spreading evenly - **backoffLimit was set far below the default for a workload with expected transient failures** _(common)_ - Someone set backoffLimit to 0 or 1 to fail fast, and a single flaky network call or a one-off eviction now kills the whole Job. The default is 6; a manifest that overrides it downward turns any transient error into a permanent Job failure. - How to tell: kubectl get job -o jsonpath='{.spec.backoffLimit}' prints 0 or 1 and .status.failed equals it exactly - **restartPolicy OnFailure hides the evidence and skews the counting** _(edge)_ - With restartPolicy OnFailure the kubelet restarts the container inside the same pod, and the Job counts those container retries in Pending or Running pods in addition to whole failed pods. When the limit is reached the pod running the job is terminated, taking its logs with it — the Job looks like it failed with no trace. - How to tell: The Job's pods show RESTARTS greater than 0 before disappearing, and kubectl logs on them returns nothing after the Job fails ## Solution 1. Confirm what actually fired. The Failed condition's reason distinguishes the failure-count limit (BackoffLimitExceeded) from the wall-clock deadline (DeadlineExceeded) — they need different fixes. ```bash kubectl describe job -n ``` Note: status.failed in kubectl get job -o yaml shows how many pod failures were counted against the limit. 2. Read the logs of the failed pods. They are not deleted when the Job fails — with restartPolicy Never every attempt leaves its own Failed pod behind, and the real error is in there, not in the Job status. ```bash kubectl get pods -l job-name= -n ``` Note: Then kubectl logs on each Failed pod. Identical output across attempts means a deterministic bug; varying output points at the environment. 3. If the pods are gone, you are running restartPolicy OnFailure — the pod is terminated once the limit is reached. Switch to Never while debugging so the evidence survives, exactly as the Kubernetes docs suggest. 4. Fix the underlying cause, then delete and re-create the Job. A Job that hit its backoff limit is permanently failed — there is no automatic restart, and most of the spec is immutable, so editing it in place is not an option. ```bash kubectl delete job -n && kubectl apply -f job.yaml ``` 5. Stop paying for retries that cannot succeed and for disruptions that are not failures: add a pod failure policy that fails the Job immediately on a non-retriable exit code and ignores pod disruptions instead of counting them. ```yaml apiVersion: batch/v1 kind: Job spec: backoffLimit: 6 podFailurePolicy: rules: - action: FailJob # a bug: fail now, don't burn six retries onExitCodes: containerName: main operator: In values: [42] - action: Ignore # node drains don't consume the budget onPodConditions: - type: DisruptionTarget template: spec: restartPolicy: Never # podFailurePolicy requires Never containers: - name: main image: my-batch:1.4 ``` Note: podFailurePolicy is stable since Kubernetes 1.31 and cannot be combined with restartPolicy OnFailure. 6. Bound the Job in wall-clock time with activeDeadlineSeconds where total duration matters more than attempt count — but know that it takes precedence over backoffLimit and kills the Job with reason DeadlineExceeded even mid-retry. 7. Choose backoffLimit deliberately rather than reflexively raising it. More retries of a deterministic failure only delay the same outcome — the delay between attempts doubles each time, capped at six minutes — while genuinely transient failures are better excluded from the count with an Ignore rule. **Verify:** The re-created Job reaches the Complete condition: kubectl get job shows COMPLETIONS satisfied, status.conditions contains type Complete instead of Failed, and status.failed stays at zero (or safely below the limit) across a full run. **If that fails:** For Indexed Jobs where one bad index should not kill the rest, set backoffLimitPerIndex so failure accounting is per index. Where the application manages its own retries idempotently, set backoffLimit to 0 and treat any pod failure as final rather than letting two retry layers multiply. ## Applies to - Kubernetes: 1.21 and later — BackoffLimitExceeded is the reason on the Job's Failed condition, set by the Job controller; the message text comes from the controller source. - Job controller (kube-controller-manager): 1.21 and later — Counts whole failed pods, plus container retries inside Pending/Running pods when restartPolicy is OnFailure. - Platforms: self-hosted, managed Kubernetes ## Not applicable to - CrashLoopBackOff on Deployments and other long-running workloads, where the kubelet restarts containers indefinitely and nothing is ever marked permanently Failed — backoffLimit is a Job concept - CronJob schedule problems — missed schedules, startingDeadlineSeconds or concurrencyPolicy skips happen before any pod runs; only the child Job a CronJob spawns can hit BackoffLimitExceeded - Jobs failed with reason DeadlineExceeded, where activeDeadlineSeconds expired — a wall-clock kill that fires even if not a single pod failed - A Job whose status.failed is still below backoffLimit — it is retrying with an exponential delay, not failed, and patience or a log read is the right move ## Evidence 1. [Jobs — Pod backoff failure policy](https://kubernetes.io/docs/concepts/workloads/controllers/job/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That backoffLimit defaults to 6, that failed pods are recreated with an exponential back-off delay capped at six minutes, that both failed pods and — under OnFailure — container retries are counted, and that activeDeadlineSeconds takes precedence over backoffLimit. > The .spec.backoffLimit is set by default to 6 ... recreated by the Job controller with an exponential back-off delay ... capped at six minutes ... .spec.activeDeadlineSeconds takes precedence over its .spec.backoffLimit 2. [Job — Kubernetes API reference (JobSpec)](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/job-v1/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That backoffLimit is the number of retries before the Job is marked failed with a default of 6, and that podFailurePolicy cannot be combined with restartPolicy=OnFailure. > Specifies the number of retries before marking this job failed. Defaults to 6, unless backoffLimitPerIndex (only Indexed Job) is specified. ... This field cannot be used in combination with restartPolicy=OnFailure. 3. [Handling retriable and non-retriable pod failures with Pod failure policy](https://kubernetes.io/docs/tasks/job/pod-failure-policy/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That a pod failure policy exists to stop retrying non-retriable failures, and that without one a Job retries to the backoffLimit — the page's own example takes at least nine minutes to fail at the default of 6. > use Pod failure policy to avoid unnecessary Pod restarts when a Pod failure indicates a non-retriable software bug ... if the Pod failure policy were disabled, the Job would retry until reaching the backoffLimit (6 failures) 4. [Pod Lifecycle — container restart policy](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) — The Kubernetes Authors (official-docs), read 2026-08-11 Supports: That OnFailure restarts the failed container in place while Never leaves the pod terminated, which is why the two policies count failures differently inside a Job and why Never preserves one pod per attempt. > OnFailure : Only restarts the container if it exits with an error (non-zero exit status). ... Jobs commonly use restartPolicy: OnFailure or restartPolicy: Never to handle batch processing tasks appropriately 5. [kubernetes/kubernetes — pkg/controller/job/job_controller.go (release-1.33)](https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.33/pkg/controller/job/job_controller.go) — The Kubernetes Authors (source-code), read 2026-08-11 Supports: That the literal condition message 'Job has reached the specified backoff limit' and the reason BackoffLimitExceeded are set by the Job controller when the failure count passes the limit. > jm.newFailureCondition(batch.JobReasonBackoffLimitExceeded, "Job has reached the specified backoff limit") ## Confidence high — The default of 6, the exponential delay and its six-minute cap, the dual counting rules, the activeDeadlineSeconds precedence, the podFailurePolicy semantics and the literal error message are all quoted from Kubernetes documentation or the Job controller source. The per-cause discriminators — same exit code across attempts, DisruptionTarget on disrupted pods, RESTARTS greater than zero under OnFailure — are diagnostic practice that follows from the documented behaviour rather than sentences any page states about diagnosis. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"kubernetes-job-backofflimitexceeded","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. When diagnosis returns an identifiedResolution, run that recipe and its verification criteria, then POST the completion body it supplies to https://knowbase.sh/outcome.json. Only a completion response with status=resolved may be presented as resolved; its receipt is agent-observed, not independently verified. The same workflow is available as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/kubernetes-job-backofflimitexceeded · knowbase · CC-BY-4.0