# knowbase — full corpus Verified, source-backed answers to concrete engineering failures. Every entry states the error, the root cause, the fix, the versions it applies to, and the primary sources that prove it. 25 knowledge objects. License CC-BY-4.0; attribute the canonical URL of each entry. --- # Container terminated with exit code 137 and reason OOMKilled Exit code 137 means the process received SIGKILL (128 + 9). In Kubernetes it almost always means the container breached its own memory limit and the cgroup OOM killer ended it — a different failure from the node running out of memory, which evicts the pod instead. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/container-exit-code-137-oomkilled ## Error signature ``` OOMKilled (exit code 137) ``` Codes: 137, OOMKilled, SIGKILL ## Problem A container is terminated abruptly with exitCode 137 and reason OOMKilled. There is no stack trace and no graceful shutdown, because SIGKILL cannot be caught or handled. If the restart policy allows it, the container comes back and is killed again at roughly the same point in its workload. ## Root cause - **The container exceeded the memory limit set on it** _(primary)_ - resources.limits.memory defines a cgroup ceiling. When the processes inside the container reach it, the kernel OOM killer terminates one of them, and the kubelet reports the container as OOMKilled with exit code 137. - How to tell: Node events show 'Memory cgroup out of memory' naming the killed process - **The runtime's heap is not aware of the cgroup limit** _(common)_ - A JVM, Node.js, or .NET process that sizes its heap from total host memory will happily grow past a much smaller container limit. The application believes it has headroom that the cgroup does not grant. - How to tell: Host memory greatly exceeds the container limit, and the process dies near a heap size close to a fraction of host RAM rather than of the limit - **A genuine memory leak or unbounded buffer** _(common)_ - Memory grows monotonically with uptime or with request volume, so raising the limit only moves the time of death later. - How to tell: Time-to-kill scales with the limit rather than being fixed at a workload step - **A sidecar consumed the shared budget** _(edge)_ - Limits are per container, but several containers in one pod compete for node memory; a log shipper or service mesh proxy under-provisioned alongside the app can be the one killed. - How to tell: The OOMKilled container is not the application container - **Node-level memory pressure rather than a container limit** _(edge)_ - When the node itself runs out of allocatable memory, the kubelet evicts pods by QoS class. This is eviction, not a cgroup OOM kill, and it is fixed by requests and node sizing rather than by limits. - How to tell: Pod status is Failed with reason Evicted and a MemoryPressure node condition ## Solution 1. Confirm the kill was a cgroup OOM and read the exact limit that was breached. ```bash kubectl describe pod -n | grep -A6 'Last State' ``` Note: Expect reason OOMKilled and exitCode 137 under lastState.terminated. 2. Compare real usage against the limit before changing it, so the new number is measured rather than guessed. ```bash kubectl top pod -n --containers ``` 3. Tell the runtime about the container limit instead of raising the limit blindly. This is the fix whenever the process sizes its heap from host memory. ```yaml # JVM: honour the cgroup limit and take a defined share of it env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=75.0" # Node.js: cap old-space below the container limit env: - name: NODE_OPTIONS value: "--max-old-space-size=768" # for a 1Gi limit ``` 4. Set requests and limits deliberately. A request equal to the limit gives the pod Guaranteed QoS and makes it the last candidate for eviction under node pressure. ```yaml resources: requests: memory: "1Gi" limits: memory: "1Gi" ``` 5. If usage grows with uptime, treat it as a leak and profile it. Raising the limit on a leaking process buys time, not a fix. 6. If the pod was evicted rather than OOMKilled, size the node or lower requests; container limits are not the lever. **Verify:** Run the workload at peak for at least one full duty cycle and confirm no lastState.terminated.reason of OOMKilled appears, with kubectl top showing steady memory below the limit rather than a sawtooth ending in a restart. **If that fails:** Where peak memory is genuinely spiky and short-lived, keep the request at the steady state and raise only the limit, accepting Burstable QoS and the eviction risk that comes with it. ## Applies to - Kubernetes: 1.20 and later — Applies wherever resources.limits.memory is set on a container. - Docker Engine: all supported versions — Same signal semantics: the runtime reports 128 + signal, so a SIGKILL surfaces as 137 here too. - Linux kernel cgroups: cgroup v1 and v2 - Runtimes: containerd, CRI-O, Docker - Platforms: linux/amd64, linux/arm64 ## Not applicable to - Exit code 143, which is SIGTERM and normally indicates a graceful shutdown or rollout - Exit code 1 or other application-defined codes, which are the program's own failure - Pods with reason Evicted, which is kubelet node-pressure eviction and not a cgroup OOM kill - Windows containers, where job-object memory limits do not produce POSIX signal exit codes ## Evidence 1. [Assign Memory Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That a container exceeding its memory limit is terminated with exitCode 137 and reason OOMKilled, that the kubelet restarts it, and the node event text emitted by the cgroup OOM killer. > exitCode: 137 ... reason: OOMKilled 2. [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: The distinction between requests and limits, and that memory limits are enforced reactively by the kernel OOM killer rather than as a pre-emptive ceiling — a container can exceed its limit and is killed once the kernel sees memory pressure. > memory limits are enforced by the kernel with out of memory (OOM) kills 3. [Node-pressure Eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That node-level memory exhaustion causes kubelet eviction by QoS class, which is a different failure mode from a per-container OOM kill. > the kubelet can proactively fail one or more pods on the node to reclaim resources and prevent starvation 4. [Bash Reference Manual — Exit Status](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html) — GNU Project (specification), read 2026-08-07 Supports: The 128 + signal-number convention that makes 137 the status of a process killed by SIGKILL (signal 9), and 143 the status of one terminated by SIGTERM. > Bash uses the value 128+N as the exit status ## Confidence high — The exit code, the reason string, and the kubelet behaviour are quoted from a Kubernetes task page that shows the literal status block, and the eviction contrast is taken from the eviction reference. The runtime heap flags are widely used defaults rather than values any single document prescribes, so they are presented as configuration examples. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"container-exit-code-137-oomkilled","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/container-exit-code-137-oomkilled · knowbase · CC-BY-4.0 --- # CORS error: No Access-Control-Allow-Origin header on the response The browser blocked the response because the server did not say the requesting origin is allowed. Nothing about the frontend can fix it — the header has to come from the server, and the browser deliberately hides the reason from JavaScript, so the console is the only place the actual rule violation is named. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/cors-no-access-control-allow-origin ## Error signature ``` No 'Access-Control-Allow-Origin' header is present on the requested resource ``` Codes: CORS, ERR_FAILED ## Problem A fetch or XMLHttpRequest to another origin fails in the browser while the same URL works from curl or Postman. The JavaScript error object says only that the request failed; it carries no status, no body and no reason. Developers conclude the API is down or their client code is wrong, when the request usually reached the server and was answered — the browser simply refused to hand the response to the page. ## Root cause - **The server never sends an Access-Control-Allow-Origin header** _(primary)_ - The endpoint was written for same-origin use, or CORS middleware is registered after the route that answers, so the header never reaches the response. The request succeeds server-side and is discarded by the browser. - How to tell: curl with an explicit Origin request header returns 200 but the response has no access-control-allow-origin header - **The preflight OPTIONS request is not handled** _(common)_ - Any request that is not a simple one — a custom header, a JSON content type, a method other than GET/HEAD/POST — makes the browser send an OPTIONS request first. Frameworks that only route GET and POST answer it with 404 or 405, and the real request is never sent. - How to tell: DevTools Network shows an OPTIONS entry before the failing request, and it returns 404, 405, or a 2xx without CORS headers - **Credentials are used together with the wildcard origin** _(common)_ - A response of Access-Control-Allow-Origin '*' is rejected outright when the request carries cookies or HTTP auth. The server must echo the specific origin instead, and add Access-Control-Allow-Credentials. - How to tell: The console message mentions credentials mode 'include' while the response header is the literal asterisk - **The preflight succeeds but does not allow the header or method being used** _(common)_ - Access-Control-Allow-Origin alone is not enough. A custom header such as Authorization or X-Request-Id must also appear in Access-Control-Allow-Headers, and non-simple methods in Access-Control-Allow-Methods. - How to tell: The console names a specific field, for example 'Request header field authorization is not allowed by Access-Control-Allow-Headers' - **Only error responses lack the header** _(edge)_ - CORS middleware often runs inside the normal request pipeline, so 500s raised earlier and 404s from the router bypass it. The API looks fine until something fails, and then the real status code is invisible to the client. - How to tell: Successful calls work and only failing ones report a CORS error, hiding the underlying 4xx or 5xx - **A proxy or CDN strips or overwrites the header** _(edge)_ - An API gateway, reverse proxy or CDN layer rewrites response headers, so the origin server sends a correct header that never reaches the browser. - How to tell: curl against the origin shows the header, curl against the public hostname does not ## Solution 1. Read the exact message in the browser console. The specification deliberately withholds the reason from JavaScript, so this text is the only place the violated rule is named — and it distinguishes a missing header from a disallowed header or method. 2. Ask the server what it actually returns for a cross-origin request. This separates a server that omits the header from a proxy that strips it. ```bash curl -si -H 'Origin: https://your-site.example' https://api.example.com/endpoint | grep -i 'access-control\|^HTTP' ``` 3. Test the preflight separately if the request is not simple. A working GET proves nothing about an OPTIONS the router never registered. ```bash curl -si -X OPTIONS -H 'Origin: https://your-site.example' -H 'Access-Control-Request-Method: POST' -H 'Access-Control-Request-Headers: authorization,content-type' https://api.example.com/endpoint ``` 4. Add the headers on the server, echoing the caller's origin rather than hardcoding one, and register the middleware before the routes so error responses carry it too. ```javascript // Express — mount before any route so 404s and 500s are covered as well const ALLOWED = new Set(["https://your-site.example"]); app.use((req, res, next) => { const origin = req.headers.origin; if (origin && ALLOWED.has(origin)) { res.setHeader("Access-Control-Allow-Origin", origin); res.setHeader("Vary", "Origin"); res.setHeader("Access-Control-Allow-Credentials", "true"); res.setHeader("Access-Control-Allow-Headers", "authorization,content-type"); res.setHeader("Access-Control-Allow-Methods", "GET,POST,PUT,DELETE,OPTIONS"); } if (req.method === "OPTIONS") return res.sendStatus(204); next(); }); ``` Note: Vary Origin matters as soon as the value is dynamic — without it a shared cache can serve one origin's allowed response to another origin. 5. If the request sends cookies or an Authorization header, never use the wildcard. Echo the exact origin and set Access-Control-Allow-Credentials to true, as above. 6. Re-run the curl checks and confirm both the preflight and the real request carry the headers, then retry from the browser with the cache disabled — preflight results are cached and a stale one keeps failing after the fix. **Verify:** curl with an Origin header shows access-control-allow-origin echoing your origin on both the OPTIONS and the real request, and the browser console reports no CORS error on a hard reload. **If that fails:** Where the API is third-party and cannot be changed, route the call through your own backend or a same-origin path rewrite. CORS is enforced by the browser, so a server-to-server request is not subject to it. ## Applies to - Browsers: all current browsers — CORS is enforced by the browser; it is not a server-side security control. - Fetch API and XMLHttpRequest: all versions — Both follow the same-origin policy and the same CORS rules. - Platforms: web ## Not applicable to - Same-origin requests, which are never subject to CORS regardless of headers - Server-side HTTP clients such as curl, Postman, or fetch inside Node.js, none of which enforce CORS - 401 and 403 responses, which are authentication or authorisation failures the browser will surface normally once CORS headers are present - Mixed-content blocking, where an HTTPS page requests an HTTP resource ## Evidence 1. [Cross-Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS) — MDN Web Docs (official-docs), read 2026-08-07 Supports: That CORS is a header-based mechanism enforced by the browser, that non-simple requests are preflighted with OPTIONS, and — critically for diagnosis — that the failure reason is withheld from JavaScript and available only in the console. > CORS failures result in errors but for security reasons, specifics about the error are not available to JavaScript. 2. [Reason: CORS header 'Access-Control-Allow-Origin' missing](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS/Errors/CORSMissingAllowOrigin) — MDN Web Docs (official-docs), read 2026-08-07 Supports: That this specific error means the response lacked the required header, rather than the request having failed to reach the server. > The response to the CORS request is missing the required Access-Control-Allow-Origin header 3. [Access-Control-Allow-Origin header reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Access-Control-Allow-Origin) — MDN Web Docs (official-docs), read 2026-08-07 Supports: That the wildcard is rejected for credentialed requests, that a server serving multiple origins must echo the requesting origin, and that a dynamic value requires Vary Origin so caches do not cross-serve responses. > Attempting to use the wildcard with credentials results in an error ## Confidence high — The mechanism, the preflight rules, the credentials restriction and the Vary requirement are all quoted from MDN's CORS reference. All three sources share a publisher, which is a real limitation — but MDN is the reference implementation documentation for browser behaviour here, and the Fetch Standard it summarises defines the same rules. The per-cause discriminators are observed console and DevTools output rather than documented diagnostic procedure. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"cors-no-access-control-allow-origin","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/cors-no-access-control-allow-origin · knowbase · CC-BY-4.0 --- # Docker error: no space left on device The disk holding Docker's data directory is full, and the space is almost never where people look first. Stopped containers, dangling images and — most often — the build cache accumulate silently, and none of them appear in a directory listing of your project. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/docker-no-space-left-on-device ## Error signature ``` no space left on device ``` Codes: ENOSPC ## Problem A build or pull fails saying the device is full, but df on the project directory shows free space. CI agents fail after weeks of working, and clearing the workspace changes nothing. The consumed space belongs to the Docker daemon rather than to any project tree, so ordinary cleanup misses all of it. ## Root cause - **The build cache has grown without bound** _(primary)_ - BuildKit keeps every intermediate layer it might reuse. On a machine that builds often — a CI runner especially — this is usually the largest consumer by a wide margin, and it is invisible to docker images. - How to tell: docker system df shows Build Cache far larger than Images, and docker images alone does not account for the used space - **Dangling and unused images accumulate** _(primary)_ - Every rebuild of a tag leaves the previous image untagged rather than deleting it. Those layers stay on disk indefinitely because nothing removes them automatically. - How to tell: docker images -f dangling=true lists many entries, or docker system df reports a large reclaimable figure for Images - **Stopped containers and their writable layers persist** _(common)_ - A container that exited keeps its writable layer and its logs until it is removed. Runs without --rm leave one behind every time. - How to tell: docker ps -a lists far more containers than docker ps, and the Containers row in docker system df is non-trivial - **Anonymous volumes are orphaned** _(common)_ - Volumes outlive the containers that created them and are never pruned by default. Anonymous volumes from removed containers are the ones nobody remembers, and they can hold database data measured in gigabytes. - How to tell: docker volume ls shows volumes with hash-like names not referenced by any running container - **Container logs are unbounded** _(common)_ - The default json-file driver writes without rotation, so one chatty long-lived container can fill a disk on its own with no image or volume growth at all. - How to tell: Files under /var/lib/docker/containers/*/*-json.log are large, while docker system df reports little reclaimable space - **Inodes are exhausted rather than bytes** _(edge)_ - Many small files can exhaust the inode table while free space remains. The error text is identical, which sends people to the wrong measurement. - How to tell: df -i shows IUse% at or near 100 while df -h shows space available ## Solution 1. Ask Docker where its space went before reaching for df. This one command separates build cache from images, containers and volumes, and shows how much of each is reclaimable. ```bash docker system df -v ``` 2. Check whether the problem is bytes or inodes. They report the same error and need different fixes. ```bash df -h /var/lib/docker && df -i /var/lib/docker ``` 3. Clear the build cache first when that is the largest row. It is safe — the worst outcome is a slower next build. ```bash docker buildx prune -f ``` Note: Use --filter until=168h to keep recent cache while dropping anything older than a week, which is usually the right trade on a CI runner. 4. Remove unused containers, networks and images together. Read what it reports before confirming, because it does not stop at dangling images. ```bash docker system prune -a ``` Note: Without -a only dangling images go; with -a every image not used by a container is removed, including ones you will pull again tomorrow. 5. Prune volumes separately and deliberately. This is the one destructive step — an orphaned volume can still hold the only copy of some data. ```bash docker volume ls -f dangling=true && docker volume prune ``` 6. Stop the recurrence rather than repeating the cleanup. Cap log size on the daemon so no container can fill the disk, and let CI prune on a schedule. ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } } ``` Note: This is /etc/docker/daemon.json and applies to containers created after a daemon restart — existing containers keep their original settings. **Verify:** docker system df reports reclaimable space back to a small fraction of the disk, the failing build or pull completes, and both df -h and df -i show headroom on the filesystem holding /var/lib/docker. **If that fails:** If the disk is too full for the daemon to operate at all, delete the largest container log files in place to buy room for prune to run, then move Docker's data-root onto a larger volume rather than repeating the recovery. ## Applies to - Docker Engine: 20.10 and later — Data lives under /var/lib/docker by default; docker system df accounts for it. - BuildKit: all versions — Build cache is managed separately from images and is pruned with buildx prune. - Platforms: linux, macos, windows ## Not applicable to - A full disk unrelated to Docker, where /var/lib/docker is on a filesystem with space - Container memory exhaustion (OOMKilled), which concerns RAM rather than disk - Registry-side quota errors on push, which are rejected remotely rather than locally - Kubernetes node disk pressure, which triggers kubelet eviction and reports its own condition ## Evidence 1. [docker system prune](https://docs.docker.com/reference/cli/docker/system/prune/) — Docker Inc. (official-docs), read 2026-08-08 Supports: That unused containers, networks and images accumulate and are removed by an explicit prune, and that volumes are excluded unless opted into — which is why routine cleanup leaves volume space behind. > Remove all unused containers, networks, images (both dangling and unused), and optionally, volumes. 2. [docker buildx prune](https://docs.docker.com/reference/cli/docker/buildx/prune/) — Docker Inc. (official-docs), read 2026-08-08 Supports: That the build cache is a distinct store cleared by its own command, separate from images — the reason a machine can be full of cache while docker images looks small. > Clears the build cache of the selected builder. ## Confidence medium — That build cache and images are separate stores with separate prune commands, and that volumes are excluded from a default prune, are quoted from Docker's own CLI reference — and those two facts carry the primary causes. Confidence is medium rather than high because it rests on two sources: the log-rotation defaults, the inode case and the operational ordering of the cleanup steps are established practice rather than statements quoted from a primary source. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"docker-no-space-left-on-device","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/docker-no-space-left-on-device · knowbase · CC-BY-4.0 --- # JWT signature verification failed: invalid signature The token's signature did not verify against the key you supplied. The specification requires rejecting it outright — there is no partial trust — and the cause is almost always a key mismatch or an algorithm disagreement rather than a tampered token. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/jwt-signature-verification-failed ## Error signature ``` invalid signature ``` Codes: JsonWebTokenError, SignatureVerificationError ## Problem A token that the issuer produced seconds ago is rejected by the verifier. It decodes correctly — the header and claims are readable and look right — so the token is clearly not corrupt. Tokens work in one environment and fail in another, which points at configuration rather than at the token itself. ## Root cause - **The verifying key is not the one that signed** _(primary)_ - Different environments, a rotated key, or a JWKS cache holding a retired key. The token is valid and the key is valid; they simply do not belong together. - How to tell: Decoding the header shows a kid that is absent from the JWKS the verifier fetched, or the two environments use different secrets - **The algorithm expected does not match the one used** _(primary)_ - A token signed with RS256 verified as HS256, or the reverse. Accepting whatever the header declares is also a well-known vulnerability — a verifier must pin the algorithms it will accept rather than trust the token to choose. - How to tell: The header alg differs from the algorithm passed to the verify call, or the verifier accepts alg from the token instead of from configuration - **The secret or key was transformed in transit** _(common)_ - A trailing newline in an environment variable, a PEM with escaped \n that was never unescaped, or base64 decoding applied once too often. The bytes used to verify differ from the bytes used to sign even though both look correct when printed. - How to tell: The signing and verifying key material differ in length or hash, even though they appear identical on screen - **The token was modified after signing** _(common)_ - A logging pipeline that trims whitespace, a URL that lost its final characters, or a cookie truncated at a size limit. Any change to the signed input invalidates the signature, which is the mechanism working as designed. - How to tell: The token does not have exactly two period characters, or its length differs from what the issuer emitted - **The wrong key is selected from a key set** _(edge)_ - With several keys published, the verifier must select by the kid header. Picking the first key in the set works until rotation adds a second one. - How to tell: Verification succeeds against one specific key in the JWKS but the verifier is not selecting by kid - **Encoding differences in the signed input** _(edge)_ - Signature is computed over the base64url-encoded header and payload exactly as transmitted. Re-serialising the JSON before verifying — reordering keys, changing whitespace — produces different input and therefore a different signature. - How to tell: The verifier reconstructs the signing input from parsed claims rather than using the original encoded segments ## Solution 1. Decode the header without verifying, to see which key and algorithm the token actually claims. This is safe to read and decides the next step. ```bash cut -d. -f1 <<< "$TOKEN" | tr '_-' '/+' | base64 -d 2>/dev/null; echo ``` 2. Confirm the verifier is using the matching key. For asymmetric tokens, fetch the issuer's JWKS and check the kid is present. ```bash curl -s https://issuer.example.com/.well-known/jwks.json | python3 -c "import sys,json;[print(k['kid'], k['alg']) for k in json.load(sys.stdin)['keys']]" ``` 3. Pin the accepted algorithms explicitly. Never let the token's own header decide how it is verified — that is the alg-confusion attack, not merely a bug. ```javascript // 🔴 trusts the token to say how it should be checked jwt.verify(token, key); // ✅ the verifier decides jwt.verify(token, key, { algorithms: ["RS256"], issuer, audience }); ``` 4. Select the key by kid rather than taking the first one, so rotation does not break verification. ```javascript import { createRemoteJWKSet, jwtVerify } from "jose"; const JWKS = createRemoteJWKSet(new URL("https://issuer.example.com/.well-known/jwks.json")); const { payload } = await jwtVerify(token, JWKS, { issuer: "https://issuer.example.com", audience: "my-api", }); ``` Note: A remote JWKS helper caches keys and refetches when an unknown kid appears, which is what makes rotation transparent. 5. Check the key material byte for byte when the key looks right but fails. Hash both sides rather than comparing them visually. ```bash printf '%s' "$JWT_SECRET" | shasum -a 256 ``` Note: A trailing newline from a file or a secrets manager changes the hash and is invisible in logs. 6. Verify against the original encoded segments rather than re-serialising the claims. The signature covers the exact bytes that were transmitted. **Verify:** A token freshly minted by the issuer verifies in the failing environment, and a token with one character altered is rejected — proving verification is actually running rather than being bypassed. **If that fails:** During a rotation window, accept both the old and new keys by verifying against the full JWKS rather than a single key, and retire the old key only once no tokens signed with it can still be within their lifetime. ## Applies to - JSON Web Token: RFC 7519 — A JWT whose validation steps fail must be rejected as invalid input. - JSON Web Signature: RFC 7515 — Signature validation is defined by JWS; JWT inherits it. - Platforms: server, browser ## Not applicable to - Expired tokens, where the signature verifies and the exp claim is in the past - Audience or issuer mismatches, which are claim checks after a successful signature verification - Malformed tokens that fail to parse before any signature is computed - Opaque or reference tokens, which carry no signature and are validated by introspection ## Evidence 1. [RFC 7519 — JSON Web Token (JWT), Validating a JWT](https://datatracker.ietf.org/doc/html/rfc7519) — IETF (specification), read 2026-08-08 Supports: That a JWT failing any validation step must be rejected and treated as invalid input — there is no degraded acceptance, which is why a key or algorithm mismatch is fatal rather than a warning. > If any of the listed steps fail, then the JWT MUST be rejected -- that is, treated by the application as an invalid input. 2. [RFC 7515 — JSON Web Signature (JWS), Message Signature or MAC Validation](https://datatracker.ietf.org/doc/html/rfc7515) — IETF (specification), read 2026-08-08 Supports: That signature validation is a defined sequence and that failure at any step means the signature cannot be validated — the mechanism JWT relies on for integrity. > If any of the listed steps fails, then the signature or MAC cannot be validated. ## Confidence medium — The normative requirement to reject on any validation failure is quoted from both relevant RFCs, which establishes the severity and the mechanism. Confidence is medium rather than high because it rests on two sources: the specific causes — key rotation, algorithm pinning against alg confusion, key-material whitespace — are drawn from well-established security practice rather than quoted from a primary source here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"jwt-signature-verification-failed","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/jwt-signature-verification-failed · knowbase · CC-BY-4.0 --- # Kubernetes pod fails with CreateContainerConfigError The image pulled fine, but the kubelet cannot assemble the container's configuration: a referenced ConfigMap or Secret does not exist, or exists without the key the pod asks for. The pod stays in Waiting and no container is ever created. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-createcontainerconfigerror ## Error signature ``` CreateContainerConfigError ``` Codes: CreateContainerConfigError ## Problem A pod sits in Waiting with reason CreateContainerConfigError. The image was pulled successfully, so this is not a registry problem, and there are no logs because no container was ever created. The kubelet retries indefinitely without progressing, because nothing about the pod spec changes on its own. ## Root cause - **The referenced ConfigMap or Secret does not exist** _(primary)_ - A required reference to an object that was never created, was deleted, or was renamed. Kubernetes refuses to start the container rather than starting it with missing configuration. - How to tell: Events read 'configmap "" not found' or 'secret "" not found', and kubectl get on that object returns NotFound - **The object exists but the referenced key does not** _(common)_ - configMapKeyRef or secretKeyRef names a key that is absent — often a typo, or a key renamed in the ConfigMap without updating the consumers. - How to tell: Events read 'couldn't find key in ConfigMap /' - **The object lives in a different namespace** _(common)_ - ConfigMaps and Secrets are namespaced and cannot be referenced across namespaces. A manifest that works in one namespace fails silently in another when the object was only ever created in the first. - How to tell: kubectl get configmap -A shows the object present, but under a namespace other than the pod's - **The pod was created before the object it depends on** _(common)_ - Applying a manifest directory in the wrong order, or a CI job that creates the Deployment before the ConfigMap. The kubelet keeps retrying, and once the object appears an env-var reference still will not pick it up until the pod is replaced. - How to tell: The object now exists and was created after the pod, yet the pod is still failing - **The reference is genuinely optional but not marked as such** _(edge)_ - A pod that should tolerate absent configuration will still refuse to start unless the reference carries optional true. - How to tell: Adding optional: true to the reference lets the pod start with empty configuration ## Solution 1. Read the pod's events. The kubelet names the exact object and, when the object exists, the exact key it could not find. ```bash kubectl describe pod -n ``` 2. Confirm whether the object exists in the pod's own namespace. Searching all namespaces at once catches the most common mistake. ```bash kubectl get configmap,secret -A | grep ``` 3. If the object exists, list its keys and compare them against what the pod asks for. A key that differs by one character fails exactly like a missing object. ```bash kubectl get configmap -n -o jsonpath='{.data}' | tr ',' '\n' ``` 4. Create whatever is missing in the correct namespace. ```bash kubectl create configmap app-config \ --from-literal=LOG_LEVEL=info \ -n ``` 5. Where the configuration really is optional, say so explicitly rather than relying on the object always being present. ```yaml env: - name: LOG_LEVEL valueFrom: configMapKeyRef: name: app-config key: LOG_LEVEL optional: true ``` Note: With optional true and the object absent, the variable is simply unset rather than blocking the container. 6. Replace the pod after fixing the reference. Creating the missing object is not enough on its own for environment-variable references, which are resolved once at container creation. ```bash kubectl rollout restart deployment/ -n ``` **Verify:** kubectl get pod shows Running, and kubectl describe pod no longer lists a Failed event mentioning the ConfigMap or Secret. **If that fails:** If the object is created by another controller or an external secrets operator that has not reconciled yet, mark the reference optional so the pod can start degraded, and have the application fail its readiness probe until the configuration arrives. ## Applies to - Kubernetes: 1.20 and later — CreateContainerConfigError is a Waiting-state reason: the pod phase stays Pending and no container is created. - kubelet: 1.20 and later - Runtimes: containerd, CRI-O ## Not applicable to - CrashLoopBackOff, where configuration resolved and the container started before exiting - ImagePullBackOff, which fails earlier, at the image pull, before configuration is read - CreateContainerError, which is a container-runtime failure rather than a missing config reference - RBAC errors on the ServiceAccount, which surface as API request failures inside the running application ## Evidence 1. [Configure a Pod to Use a ConfigMap — Restrictions](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That a ConfigMap must exist before it is referenced, that an unmarked reference to a missing ConfigMap stops the pod starting, and that a missing key behaves the same way unless the reference is optional. > You must create the ConfigMap object before you reference it in a Pod specification. 2. [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That a Secret must exist before the pods depending on it, and that when the kubelet cannot fetch one it retries and records an Event describing the problem — which is why the pod loops without progressing. > a Secret needs to be created before any Pods that depend on it 3. [Debug Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That describing the pod and reading its recent events is the first diagnostic step, which is where the missing object and key are named. > The first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events ## Confidence high — The creation-order requirement, the optional-reference semantics and the retry behaviour are stated directly in Kubernetes' ConfigMap and Secret documentation. The event strings used as discriminators are the kubelet's own messages rather than quoted documentation, and the namespace-mismatch case follows from ConfigMaps being namespaced rather than from a sentence about this specific failure. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"kubernetes-createcontainerconfigerror","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/kubernetes-createcontainerconfigerror · knowbase · CC-BY-4.0 --- # Kubernetes pod stuck in CrashLoopBackOff CrashLoopBackOff is not an error in itself — it is the kubelet waiting between restarts of a container that keeps exiting. The exit code and reason on the previous container instance identify the actual failure; the backoff only controls how long you wait to see it again. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-crashloopbackoff ## Error signature ``` CrashLoopBackOff ``` Codes: CrashLoopBackOff, BackOff ## Problem A pod reports STATUS CrashLoopBackOff and its restart count climbs. The container starts, exits, and the kubelet restarts it after an increasing delay. Logs from the running container are empty or unavailable because the process has already died by the time you look. ## Root cause - **The container process exits non-zero during startup** _(primary)_ - Missing configuration, an unreachable dependency at boot, a failed migration, or an unhandled startup exception. The application decides to die; Kubernetes only reports it. - How to tell: lastState.terminated.exitCode is non-zero and reason is Error - **The container is killed for exceeding its memory limit** _(common)_ - The cgroup OOM killer terminates the process, the kubelet restarts it, and it dies again at the same allocation point. - How to tell: lastState.terminated.reason is OOMKilled with exitCode 137 - **A failing liveness probe restarts a container that is otherwise running** _(common)_ - A probe that is too aggressive, points at the wrong port or path, or fires before a slow application finishes booting will restart a healthy process indefinitely. A startupProbe is the correct fix for slow boots, not a longer liveness period. - How to tell: Events show 'Liveness probe failed' and 'Container failed liveness probe, will be restarted' - **Invalid command, args, or entrypoint** _(common)_ - An overridden `command` that does not exist in the image, or arguments the binary rejects, produces an immediate non-zero exit. - How to tell: exitCode 126 (not executable) or 127 (not found), or the log shows a usage message - **The entrypoint completes successfully but restartPolicy is Always** _(edge)_ - A one-shot process that exits 0 is restarted forever under restartPolicy Always. Such workloads belong in a Job, not a Deployment. - How to tell: lastState.terminated.exitCode is 0 and reason is Completed ## Solution 1. Read the previous container's termination state. This, not the pod status, names the failure. ```bash kubectl describe pod -n ``` Note: Look at Last State, Exit Code, Reason, and the Events list at the bottom. 2. Read the logs of the instance that already died. Without --previous you get the logs of the container that is currently waiting to start, which are empty. ```bash kubectl logs -n --previous ``` 3. Map the exit code to a cause before changing anything. 137 means SIGKILL, almost always the memory limit; 143 means SIGTERM; 126 and 127 point at the entrypoint; anything else is the application's own exit status. 4. If Events show a liveness probe failure, add or widen a startupProbe rather than loosening liveness. A startupProbe suspends liveness checks until the application reports ready. ```yaml startupProbe: httpGet: { path: /healthz, port: 8080 } failureThreshold: 30 periodSeconds: 5 # allows up to 150s to boot livenessProbe: httpGet: { path: /healthz, port: 8080 } periodSeconds: 10 ``` 5. Fix the underlying defect and redeploy. Deleting the pod does not reset the backoff decision, it only reschedules the same failing spec. ```bash kubectl rollout restart deployment/ -n ``` 6. While iterating, expect long gaps between restarts. The kubelet backs off starting at 10 seconds, doubles each failure, and caps at 5 minutes. The counter resets only after the container has run successfully for 10 minutes. Note: Clusters with the ReduceDefaultCrashLoopBackoffDecay feature gate enabled start at 1 second and cap at 1 minute instead. **Verify:** kubectl get pod -n shows STATUS Running with a stable RESTARTS count over at least 10 minutes, which is also the window after which the kubelet resets its backoff timer. **If that fails:** If the container dies before producing any logs, override the entrypoint with a sleep and exec into it to inspect the environment from inside — for example set `command: ["sleep", "3600"]`, then `kubectl exec -it -- sh`. ## Applies to - Kubernetes: 1.20 and later — CrashLoopBackOff is surfaced by kubectl and is not a field of the Pod API; the Pod phase stays Running while the container waits. - kubelet: 1.20 and later — Backoff defaults of 10s initial and 300s maximum apply through 1.31; from 1.32 the ReduceDefaultCrashLoopBackoffDecay gate offers 1s initial and 60s maximum. - Runtimes: containerd, CRI-O - Platforms: linux/amd64, linux/arm64 ## Not applicable to - ImagePullBackOff or ErrImagePull, which are registry or credential failures and never start the container - CreateContainerConfigError, which means a referenced ConfigMap or Secret key is missing - Pods stuck in Pending, which is a scheduling problem, not a restart loop - Init container failures, which report Init:CrashLoopBackOff and must be debugged against the init container name ## Evidence 1. [Pod Lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That CrashLoopBackOff is a kubectl-surfaced status rather than part of the Pod API data model, and the restartPolicy semantics behind repeated restarts. > CrashLoopBackOff may appear in the Status field of some kubectl commands. 2. [Debug Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: The describe-then-logs diagnostic sequence and reading the terminated state of the previous container instance. > The first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events 3. [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That a failing liveness probe causes the kubelet to restart the container, and that startupProbe is the mechanism for slow-starting applications. > Kubernetes provides liveness probes to detect and remedy such situations 4. [KEP-4603: Tune CrashLoopBackoff](https://raw.githubusercontent.com/kubernetes/enhancements/master/keps/sig-node/4603-tune-crashloopbackoff/README.md) — Kubernetes Enhancements (specification), read 2026-08-07 Supports: The exact backoff values: 10s initial with 2x decay capped at five minutes and a 10-minute success reset today, versus 1s initial capped at 1 minute behind the ReduceDefaultCrashLoopBackoffDecay gate. > that is capped at five minutes. The delay for restarts will stay at 5 minutes until a container has executed for 2x the maximum backoff ## Confidence high — Diagnostic steps and probe semantics come from Kubernetes' own documentation, and the backoff timings come from the enhancement proposal that defines them rather than from folklore. The stable-release target for the reduced-decay gate is a plan, not a shipped fact, and is described here as such. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"kubernetes-crashloopbackoff","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/kubernetes-crashloopbackoff · knowbase · CC-BY-4.0 --- # Kubernetes pod stuck in ImagePullBackOff or ErrImagePull The kubelet cannot pull the image, so the container never starts. ErrImagePull is the first failure and ImagePullBackOff is the wait between retries — neither says why. The registry's own error text, printed in the pod's Events, is what identifies the cause. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-imagepullbackoff ## Error signature ``` ImagePullBackOff ``` Codes: ImagePullBackOff, ErrImagePull ## Problem A pod never reaches Running. Its status alternates between ErrImagePull and ImagePullBackOff, and the restart interval grows. kubectl logs returns nothing because no container was ever created — the failure happens before the image becomes a container, so the usual log-reading reflex gives you nothing to work with. ## Root cause - **The image reference is wrong or the tag does not exist** _(primary)_ - A typo in the repository, a tag that was never pushed, or a tag that has since been deleted or overwritten. The registry answers correctly; the reference is simply wrong. - How to tell: Events show 'manifest unknown' or 'not found', and pulling the exact same reference from your own machine fails identically - **The registry is private and credentials are missing or wrong** _(common)_ - No imagePullSecrets on the pod or its ServiceAccount, a secret in the wrong namespace, or credentials that have expired. Image pull secrets are namespaced, so one that works in staging does nothing in production. - How to tell: Events show '401 Unauthorized', 'pull access denied' or 'authentication required' - **The registry rate-limited the pull** _(common)_ - Docker Hub limits unauthenticated pulls to 100 per IPv4 address or IPv6 /64 subnet per six hours. A cluster behind one NAT address shares that budget across every node, so a busy cluster exhausts it without any single workload looking unusual. - How to tell: Events contain 'toomanyrequests' or 'You have reached your pull rate limit' - **The node cannot reach the registry** _(common)_ - DNS failure, an egress firewall, a proxy that needs configuring on the container runtime rather than the pod, or a private registry with no route from the node subnet. - How to tell: Events show 'dial tcp ... i/o timeout', 'no such host' or a TLS handshake failure - **The image has no build for the node's architecture** _(edge)_ - An amd64-only image scheduled onto an arm64 node. The reference resolves and credentials work, but the manifest list has no matching entry. - How to tell: Events show 'no matching manifest for linux/arm64' or a similar platform string ## Solution 1. Read the Events at the bottom of the pod description. The registry's own error text is there, and it names the cause; the pod status never does. ```bash kubectl describe pod -n ``` Note: Look for the Failed event from the kubelet — it quotes the runtime's message verbatim. 2. Reproduce the pull from outside the cluster with the exact same reference. This separates a wrong reference from a cluster-side problem in one step. ```bash crane manifest /: || docker pull /: ``` 3. If the registry is private, create a pull secret and attach it to the pod in the same namespace as the pod. ```bash kubectl create secret docker-registry regcred \ --docker-server= \ --docker-username= \ --docker-password= \ -n ``` Note: Then reference it under spec.imagePullSecrets, or attach it to the ServiceAccount so every pod in the namespace inherits it. 4. Wire the secret into the pod spec, or the kubelet will keep pulling anonymously. ```yaml spec: imagePullSecrets: - name: regcred containers: - name: app image: /: ``` 5. If the message mentions rate limiting, authenticate rather than retry. Unauthenticated pulls share a per-IP budget across the whole cluster, so retrying makes it worse. Mirroring hot images into your own registry removes the dependency entirely. 6. Fix the reference or the credentials, then replace the pod. Editing nothing and waiting does not help — the kubelet retries with the same failing spec, backing off up to five minutes between attempts. ```bash kubectl rollout restart deployment/ -n ``` **Verify:** kubectl describe pod shows a 'Successfully pulled image' event and the pod reaches Running; kubectl get pod reports a stable RESTARTS count. **If that fails:** Where the registry is genuinely unreachable from the cluster — an air-gapped environment, for example — pre-load the image onto the nodes and set imagePullPolicy to IfNotPresent so the kubelet uses the local copy. ## Applies to - Kubernetes: 1.20 and later — ImagePullBackOff is a Waiting-state reason surfaced by the kubelet, not a Pod phase. - kubelet: 1.20 and later — Pull retry delay grows to a compiled-in ceiling of 300 seconds. - Runtimes: containerd, CRI-O - Platforms: linux/amd64, linux/arm64 ## Not applicable to - CrashLoopBackOff, where the image pulled successfully and the container starts then exits - CreateContainerConfigError, where a referenced ConfigMap or Secret is missing - ErrImageNeverPull, which means imagePullPolicy is Never and the image is absent from the node - Pods stuck in Pending, which is a scheduling problem and never reaches the pull stage ## Evidence 1. [Images — ImagePullBackOff](https://kubernetes.io/docs/concepts/containers/images/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: The definition of ImagePullBackOff, that an invalid image name and a private registry without an imagePullSecret are its named causes, and that the retry delay grows to a compiled-in ceiling of 300 seconds. > The status ImagePullBackOff means that a container could not start because Kubernetes could not pull a container image ... which is 300 seconds (5 minutes) 2. [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That a docker-registry Secret is the mechanism for authenticated pulls and that it is referenced from the pod through imagePullSecrets. > Create a Secret by providing credentials on the command line 3. [Docker Hub pull usage and limits](https://docs.docker.com/docker-hub/usage/pulls/) — Docker Inc. (official-docs), read 2026-08-07 Supports: The unauthenticated pull ceiling of 100 per IPv4 address or IPv6 /64 subnet per six hours, which is why a NATed cluster exhausts the budget collectively. > Unauthenticated and Docker Personal users are subject to a 6-hour pull rate limit on Docker Hub. 4. [Debug Pods](https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/) — The Kubernetes Authors (official-docs), read 2026-08-07 Supports: That describing the pod and reading its recent events is the first diagnostic step, which is where the registry's own error text appears. > The first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events ## Confidence high — The status definition, the retry ceiling and the named causes come from Kubernetes' own image documentation, and the pull-limit figures from Docker's published table. The per-cause discriminators are the runtime's own error strings as surfaced in pod events rather than claims any single document makes about diagnosis. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"kubernetes-imagepullbackoff","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/kubernetes-imagepullbackoff · knowbase · CC-BY-4.0 --- # Kubernetes pod stuck in Pending: no nodes are available The scheduler could not find a node that satisfies the pod, so it stays Pending indefinitely. The Events line enumerates exactly why each node was rejected — and that reason, not the Pending status, is the whole diagnosis. > Confidence: high · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/kubernetes-pod-pending-insufficient-resources ## Error signature ``` 0/3 nodes are available: Insufficient cpu ``` Codes: Pending, FailedScheduling, Unschedulable ## Problem A pod is created but never starts. It shows Pending with no container status and no logs, because no node has accepted it yet. Nothing is crashing and nothing is retrying in the usual sense — the scheduler is simply unable to place it, and will keep the pod queued until the cluster changes. ## Root cause - **No node has enough unreserved CPU or memory for the requests** _(primary)_ - Scheduling is decided by resources.requests, not by actual usage. A node running at 20% CPU can still be unable to accept a pod if existing requests already reserve its capacity — the scheduler counts promises, not consumption. - How to tell: Events say 'Insufficient cpu' or 'Insufficient memory', and kubectl describe node shows Allocated resources requests near 100% despite low real usage - **A node selector or affinity rule matches nothing** _(common)_ - nodeSelector or requiredDuringSchedulingIgnoredDuringExecution affinity restricts the pod to labels no node carries — often a typo, or a label that exists only in another environment. - How to tell: Events say node(s) didn't match Pod's node affinity/selector, and kubectl get nodes --show-labels lacks the label the pod requires - **Every candidate node carries a taint the pod does not tolerate** _(common)_ - Control-plane nodes and dedicated pools are tainted deliberately. A pod without the matching toleration is excluded from them, which can leave nothing eligible in a small cluster. - How to tell: Events mention untolerated taint, and kubectl describe node lists a taint the pod spec has no toleration for - **A PersistentVolumeClaim cannot be bound** _(common)_ - The pod cannot be placed until its volume can be. A missing StorageClass, an exhausted quota, or a zone mismatch between the volume and the candidate nodes all hold the pod in Pending. - How to tell: Events mention unbound immediate PersistentVolumeClaims, and kubectl get pvc shows the claim Pending rather than Bound - **The cluster has no room and no autoscaler** _(common)_ - Requests genuinely exceed cluster capacity. Without a cluster autoscaler the pod waits forever; with one it waits for a node that may be blocked by quota or instance availability. - How to tell: Total requests across pending and running pods exceed cluster allocatable, and no scale-up event appears in the autoscaler's log - **Topology spread or anti-affinity forbids the remaining nodes** _(edge)_ - A rule requiring pods to spread across zones or to avoid co-location can make every node with capacity ineligible. Capacity exists; policy forbids using it. - How to tell: Events cite node(s) didn't satisfy existing pods anti-affinity rules or a topology spread constraint, while nodes still report free capacity ## Solution 1. Read the FailedScheduling event. It lists every node and the specific predicate each one failed, which identifies the cause without further guessing. ```bash kubectl describe pod -n | tail -20 ``` Note: The counts matter: '2 Insufficient cpu, 1 node(s) had untolerated taint' means two separate problems, and fixing only one leaves the pod Pending. 2. Compare what is requested against what is actually reserved on the nodes. Allocated requests, not current utilisation, is what the scheduler reads. ```bash kubectl describe nodes | grep -A6 'Allocated resources' ``` 3. If requests are simply too large, right-size them to measured usage rather than raising cluster capacity to fit a guess. ```yaml resources: requests: cpu: "250m" # what it needs to be scheduled memory: "256Mi" limits: memory: "512Mi" # ceiling, not a scheduling input ``` Note: Only requests affect scheduling. A large limit with a small request schedules easily and risks eviction later; the two answer different questions. 4. For selector or affinity failures, check the labels that actually exist before changing the rule. ```bash kubectl get nodes --show-labels ``` 5. For taints, either tolerate them explicitly or target an untainted pool. ```yaml tolerations: - key: "dedicated" operator: "Equal" value: "batch" effect: "NoSchedule" ``` 6. For an unbound claim, resolve storage first — the pod cannot be scheduled while its volume cannot be provisioned. ```bash kubectl get pvc -n && kubectl get storageclass ``` **Verify:** kubectl get pod shows the pod leaving Pending for ContainerCreating and then Running, and kubectl describe pod records a Scheduled event naming the node. **If that fails:** To get a workload running while capacity is arranged, lower its requests to the minimum it can actually start with and set a PriorityClass so it can preempt lower-priority pods rather than queueing behind them. ## Applies to - Kubernetes: 1.20 and later — Scheduling considers resources.requests only; limits play no part in node selection. - kube-scheduler: 1.20 and later — Filters nodes to feasible ones, then scores them; an empty filter result leaves the pod queued. - Platforms: linux/amd64, linux/arm64 ## Not applicable to - CrashLoopBackOff, where scheduling succeeded and the container starts then exits - ImagePullBackOff, which happens after scheduling, on the assigned node - Pods evicted under node pressure, which were scheduled and later removed - Init:0/1 status, where the pod is scheduled and an init container is still running ## Evidence 1. [Kubernetes Scheduler](https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/) — The Kubernetes Authors (official-docs), read 2026-08-08 Supports: That the scheduler filters nodes to those meeting a pod's requirements and leaves the pod unscheduled when none are suitable — which is exactly the Pending state, rather than an error condition. > If none of the nodes are suitable, the pod remains unscheduled until the scheduler is able to place it. 2. [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) — The Kubernetes Authors (official-docs), read 2026-08-08 Supports: The distinction between requests and limits, and that requests are what the scheduler reserves against node capacity — which is why a lightly loaded node can still reject a pod. > memory limits are enforced by the kernel with out of memory (OOM) kills 3. [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) — The Kubernetes Authors (official-docs), read 2026-08-08 Supports: That placement can be constrained by nodeSelector, affinity and anti-affinity — the mechanisms behind the selector-mismatch and topology causes, each of which can leave a pod unschedulable while capacity is still free. > nodeSelector field matching against node labels ## Confidence high — The filter-then-score model and the fact that an empty feasible set leaves a pod queued are quoted from the scheduler's own documentation, and the requests/limits distinction from the resource management reference. The specific event strings used as discriminators are the scheduler's own messages rather than documented text, and the PriorityClass fallback is standard practice. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"kubernetes-pod-pending-insufficient-resources","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/kubernetes-pod-pending-insufficient-resources · knowbase · CC-BY-4.0 --- # MySQL error 1205: Lock wait timeout exceeded; try restarting transaction A transaction waited 50 seconds for a row lock and gave up. This is not a deadlock — nothing is circular, someone is simply holding the lock too long. The transaction that reports the error is the victim; the one to find is the one that never committed. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/mysql-1205-lock-wait-timeout ## Error signature ``` Lock wait timeout exceeded; try restarting transaction ``` Codes: 1205, ER_LOCK_WAIT_TIMEOUT, HY000 ## Problem A write fails after a long pause rather than immediately, and retrying often succeeds. It appears under load and never in isolation. The statement in the error is the one that waited, which sends people to optimise it — while the actual problem is a different transaction that held a lock and did not release it. ## Root cause - **Another transaction holds the lock and has not committed** _(primary)_ - A transaction opened, wrote a row, and then did something slow — an HTTP call, a queue publish, waiting on user input — before committing. Its locks are held for the whole duration, and everything touching those rows queues behind it. - How to tell: information_schema.innodb_trx shows a transaction in a running state with a trx_started timestamp many seconds old and few rows modified - **A transaction was left open by the application** _(primary)_ - Autocommit disabled and no explicit commit, or an error path that returns without rollback. The connection sits idle holding locks until it is reused or closed — indefinitely, from the database's point of view. - How to tell: The blocking thread's state is idle or sleeping while its transaction is still active in innodb_trx - **The statement locks far more rows than it changes** _(common)_ - Without a usable index, InnoDB locks index records across the range it has to scan, not just the rows that match. A poorly indexed UPDATE serialises writers that would otherwise not conflict at all. - How to tell: EXPLAIN on the blocking statement shows a full scan or a large rows estimate, and adding an index reduces both the scan and the contention - **Genuine contention on the same rows** _(common)_ - A counter row, a job queue head, or a single settings row updated by every request. Every writer is correct and short; there are simply too many of them for one row. - How to tell: The waiting and blocking statements target the same primary key, and wait time scales with request rate rather than with statement duration - **The timeout is shorter than the workload needs** _(edge)_ - innodb_lock_wait_timeout defaults to 50 seconds. A batch job that legitimately holds locks for longer will time out other work regardless of how well written it is. - How to tell: The wait fails at a consistent boundary matching the configured timeout, and the blocking transaction is a known long-running job - **A schema change is holding a metadata lock** _(edge)_ - DDL takes metadata locks that block writes to the table. An ALTER waiting behind an open transaction blocks everything behind itself in turn. - How to tell: SHOW PROCESSLIST includes a thread in 'Waiting for table metadata lock', and a DDL statement is present ## Solution 1. Find who is blocking whom while it is happening. This is the whole diagnosis — the error names the victim, this names the culprit. ```bash mysql -e "SELECT waiting_pid, waiting_query, blocking_pid, blocking_query, wait_age FROM sys.innodb_lock_waits\G" ``` Note: On MySQL without the sys schema, use information_schema.innodb_trx joined against performance_schema.data_lock_waits. 2. List long-running transactions, including ones whose connection looks idle. An idle connection with an open transaction is the classic cause. ```bash mysql -e "SELECT trx_id, trx_state, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_s, trx_rows_locked, trx_query FROM information_schema.innodb_trx ORDER BY trx_started\G" ``` 3. Shorten the transaction so locks are held for as little time as possible. Do slow work before opening it, not inside it. ```javascript // 🔴 the lock is held across a network call await db.begin(); await db.query("UPDATE orders SET status='paid' WHERE id=?", [id]); await paymentProvider.capture(id); // seconds, holding the row lock await db.commit(); // ✅ external work first, transaction last and brief const receipt = await paymentProvider.capture(id); await db.begin(); await db.query("UPDATE orders SET status='paid', receipt=? WHERE id=?", [receipt, id]); await db.commit(); ``` 4. Make sure every path commits or rolls back, including error paths. A returned connection with an open transaction keeps its locks. ```javascript const conn = await pool.getConnection(); try { await conn.beginTransaction(); await work(conn); await conn.commit(); } catch (err) { await conn.rollback(); throw err; } finally { conn.release(); } ``` 5. Index the columns the blocking statement filters on, so it locks the rows it changes rather than everything it scans. ```bash mysql -e "EXPLAIN UPDATE orders SET status='x' WHERE customer_id=42;" ``` 6. Retry the victim with backoff. A lock wait timeout is transient by nature, and the statement is safe to re-run after a rollback. ```javascript for (let attempt = 0; attempt < 3; attempt++) { try { return await runTransaction(); } catch (e) { if (e.errno !== 1205 || attempt === 2) throw e; await sleep(100 * 2 ** attempt * (0.5 + Math.random())); } } ``` 7. Adjust innodb_lock_wait_timeout only after the above. Raising it makes callers wait longer rather than reducing contention; lowering it fails faster, which is sometimes the better trade for interactive requests. **Verify:** Under the load that produced them, sys.innodb_lock_waits stays empty and no 1205 reaches the application, with any residual occurrences retried transparently. **If that fails:** To clear an incident already in progress, kill the blocking transaction rather than the waiters — KILL on the blocking_pid from innodb_lock_waits releases its locks and lets the queue drain immediately. ## Applies to - MySQL: 5.7 and later — innodb_lock_wait_timeout defaults to 50 seconds and is settable per session. - InnoDB: all versions — Locks are taken on index records, so a statement without a usable index locks more rows than it modifies. - Platforms: self-hosted, managed MySQL ## Not applicable to - MySQL error 1213 deadlock found, which is a detected cycle aborted immediately rather than a wait that expired - PostgreSQL 40P01, the equivalent deadlock condition in a different engine - Connection pool acquisition timeouts, which occur before any database lock is requested - Query timeouts such as max_execution_time, where the statement is slow rather than blocked ## Evidence 1. [MySQL Server Error Reference — ER_LOCK_WAIT_TIMEOUT](https://dev.mysql.com/doc/mysql-errors/8.4/en/server-error-reference.html) — Oracle (official-docs), read 2026-08-08 Supports: That error 1205 is ER_LOCK_WAIT_TIMEOUT with SQLSTATE HY000, reported by InnoDB when a lock wait expires — distinct from the deadlock error, which is 1213. > Lock wait timeout exceeded; try restarting transaction 2. [MySQL — InnoDB Locking](https://dev.mysql.com/doc/refman/8.4/en/innodb-locking.html) — Oracle (official-docs), read 2026-08-08 Supports: That InnoDB locks index records rather than rows in the abstract, which is why a statement without a usable index locks far more than it changes and serialises writers unnecessarily. > A record lock is a lock on an index record. ## Confidence medium — The error's identity and SQLSTATE, and the fact that InnoDB locks index records, are quoted from MySQL's own reference — and the index-scope claim is the least obvious of the causes. Confidence is medium rather than high because it rests on two sources: the 50-second default and the diagnostic queries against sys and information_schema are standard practice rather than statements quoted here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"mysql-1205-lock-wait-timeout","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/mysql-1205-lock-wait-timeout · knowbase · CC-BY-4.0 --- # Next.js DynamicServerError: route couldn't be rendered statically A route used a request-time API while Next.js was prerendering it. Normally the framework catches this and quietly switches the route to dynamic rendering — so when you see the error, the call escaped the async context it was supposed to run in. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/nextjs-dynamic-server-usage ## Error signature ``` DynamicServerError: Route couldn't be rendered statically because it used ``` Codes: DynamicServerError ## Problem A build fails on a route that works in development. The message names a request-time API such as headers or cookies, but the route often does not call it directly — a library does, or the call is buried behind an await. Development never prerenders, so the failure appears only at build time, which makes it look like a build-tool problem rather than a rendering one. ## Root cause - **A request-time API is called where the route is expected to be static** _(primary)_ - Reading headers, cookies, or searchParams makes a route dependent on the request. Next.js normally responds by opting the route into dynamic rendering, so an error means it could not make that switch. - How to tell: The message names the specific API, and the route or something it imports calls headers(), cookies() or draftMode() - **The call escaped its async context** _(primary)_ - These APIs read from async context bound to the current render. Calling one inside setTimeout or setInterval runs it on a different call stack, where the context no longer exists — this is the case the framework cannot recover from. - How to tell: The call sits inside a timer, a callback, or an event handler rather than directly in the component or route handler body - **The call happens after an unawaited promise** _(common)_ - A promise that is not awaited lets execution continue past the render, so the later call lands in a fresh execution context with the original one already gone. - How to tell: An async call in the same function is not awaited, and adding await removes the error - **A third-party library reads request state** _(common)_ - An analytics, auth or feature-flag package calls headers() or cookies() internally. The route looks static; its dependency is not. - How to tell: The stack trace passes through node_modules, and removing the library's call makes the route prerender - **The route is pinned static but needs request data** _(edge)_ - An explicit force-static declaration forbids the automatic switch to dynamic rendering, turning what would have been a silent opt-in into a hard error. - How to tell: The segment exports dynamic set to force-static while also reading request state ## Solution 1. Read which API the message names and find where it is called. If it is not in your code, the stack trace will point into a dependency. ```bash npm run build 2>&1 | grep -A12 'DynamicServerError' ``` 2. Move the call out of any timer or callback and into the render path directly. This is the fix for the case the framework cannot handle for you. ```typescript // 🔴 runs on a different call stack; the context is gone setTimeout(() => { const h = headers(); }, 0); // ✅ read it where the context exists, then use the value const h = await headers(); setTimeout(() => use(h.get("x-request-id")), 0); ``` 3. Await every promise in the path leading to the call, so execution does not continue past the render that owns the context. ```typescript // 🔴 the later read happens after this render finished loadSomething(); const c = await cookies(); // ✅ await loadSomething(); const c = await cookies(); ``` 4. If the route genuinely depends on the request, declare that instead of fighting it. Dynamic rendering is a legitimate choice, not a failure. ```typescript export const dynamic = "force-dynamic"; ``` Note: This renders the route per request. Prefer it over force-static when request data is genuinely needed — pinning static and then reading the request is contradictory. 5. To keep most of the page static, isolate the request-dependent part behind a Suspense boundary so only that subtree is dynamic. ```tsx import { Suspense } from "react"; export default function Page() { return ( <> }> ); } ``` 6. For a third-party library, move its call into a Client Component or a route handler where request access is expected, rather than letting it run during prerendering. **Verify:** npm run build completes, and the build output lists the route as either static or dynamic deliberately — matching what you intended rather than whatever the framework could manage. **If that fails:** Where a dependency reads request state and cannot be isolated, mark the whole segment force-dynamic and accept per-request rendering for it, keeping sibling routes static so the cost stays contained. ## Applies to - Next.js: 13.4 and later — Applies to the App Router; the Pages Router has no equivalent prerender-time check. - React Server Components: all versions used by Next.js App Router - Runtimes: Node.js, Edge runtime ## Not applicable to - Hydration mismatches, which occur in the browser after HTML is delivered - Pages Router getStaticProps errors, which have different semantics and messages - Runtime 500s in production, which happen per request rather than during prerendering - Build failures from type or lint errors, which never reach the rendering stage ## Evidence 1. [DynamicServerError - Dynamic Server Usage](https://nextjs.org/docs/messages/dynamic-server-error) — Vercel (official-docs), read 2026-08-08 Supports: That the framework normally catches this and opts the route into dynamic rendering, that an uncaught occurrence is what surfaces as a build error, and that calls made from a timer or after an unawaited promise lose the async context the APIs depend on. > if it detects usage of a dynamic function, and catch it to automatically opt the page into dynamic rendering 2. [Next.js — Caching and revalidating, route segment config](https://nextjs.org/docs/app/guides/caching-without-cache-components) — Vercel (official-docs), read 2026-08-08 Supports: That a route segment can be pinned to dynamic rendering explicitly, and what that means — rendering per request rather than at build time. > which will result in routes being rendered for each user at request time ## Confidence medium — The automatic opt-in behaviour, the async-context requirement and the two escape scenarios are quoted from Next.js's own error documentation, which is the authority for this message. Confidence is medium rather than high because it rests on two sources: the Suspense isolation pattern and the third-party-library case are established practice rather than statements quoted from a primary source here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"nextjs-dynamic-server-usage","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/nextjs-dynamic-server-usage · knowbase · CC-BY-4.0 --- # nginx 413: Request Entity Too Large on upload nginx refused the request body because it exceeded client_max_body_size, which defaults to just 1 MB. The trap is that several layers can each impose their own limit, and raising it in the application changes nothing when the proxy in front rejects the body first. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/nginx-413-request-entity-too-large ## Error signature ``` 413 Request Entity Too Large ``` Codes: 413 ## Problem File uploads fail above a certain size while smaller ones succeed. The application never sees the request, so its own logs are silent and its upload limit appears to be ignored. Raising the framework's limit has no effect, which makes the boundary look arbitrary — it is nginx rejecting the body before anything downstream is involved. ## Root cause - **client_max_body_size is at its default of 1 MB** _(primary)_ - nginx caps request bodies at one megabyte unless told otherwise. Most people never set the directive, so the first upload over that size fails on a stock configuration. - How to tell: The nginx error log records 'client intended to send too large body' with the exact byte count, and the boundary sits at 1048576 bytes - **The directive is set in the wrong context** _(primary)_ - client_max_body_size applies in http, server and location blocks, and the most specific match wins. A generous value in http is overridden by a stale small one in the location that actually handles uploads. - How to tell: nginx -T shows the directive more than once, with a smaller value in the block matching the upload path - **Another proxy layer in front imposes its own limit** _(common)_ - A CDN, load balancer or ingress controller rejects the body before nginx sees it. The response looks identical, so fixing nginx changes nothing. - How to tell: curl against the origin accepts the upload while the same request to the public hostname fails, and nginx's own error log has no matching entry - **The Kubernetes ingress annotation is missing** _(common)_ - An ingress-nginx controller generates its configuration from annotations, so editing nginx.conf inside the pod is overwritten. The limit has to be declared on the Ingress object. - How to tell: The rejection comes from an ingress-nginx pod and no proxy-body-size annotation is present on the Ingress - **The application also has a limit, reached after nginx is fixed** _(edge)_ - Raising nginx's limit moves the failure downstream rather than removing it. The framework then rejects the body with its own error, which is easy to mistake for the fix not working. - How to tell: The status code changes from 413 to a framework-specific error, or the message now names the application rather than nginx ## Solution 1. Confirm which layer is rejecting the request. nginx logs the intended body size when it refuses, so its absence tells you the rejection happened upstream. ```bash grep 'too large body' /var/log/nginx/error.log | tail -5 ``` 2. Print the configuration nginx has actually loaded, including every include, and look for competing definitions rather than assuming the file you edited is the one in force. ```bash nginx -T 2>/dev/null | grep -n 'client_max_body_size' ``` 3. Set the limit in the block that handles uploads, and keep it consistent with any outer value so the specific one is not accidentally smaller. ```nginx http { client_max_body_size 50m; # sensible ceiling for the whole server server { location /api/upload { client_max_body_size 200m; # larger only where genuinely needed proxy_pass http://app; } } } ``` Note: A value of 0 disables the check entirely. That turns a bounded rejection into an unbounded one, so prefer a real ceiling over removing it. 4. Reload rather than restart, after testing the configuration parses. ```bash nginx -t && nginx -s reload ``` 5. On Kubernetes, set it on the Ingress. Editing the controller's generated config is undone on the next reconcile. ```yaml metadata: annotations: nginx.ingress.kubernetes.io/proxy-body-size: "200m" ``` 6. Raise the application's own limit to match, and expect the failure to reappear there once nginx stops rejecting. Both layers have to agree. ```javascript // Express app.use(express.json({ limit: "200mb" })); // PHP — php.ini // upload_max_filesize = 200M // post_max_size = 200M ``` **Verify:** An upload just under the new limit succeeds end to end, one just over it is rejected with 413 by the layer you configured, and the nginx error log names the expected boundary rather than 1048576. **If that fails:** For genuinely large files, stop proxying the body at all: issue a pre-signed URL and have the client upload directly to object storage. That removes the limit question from every layer instead of raising it in each. ## Applies to - nginx: all versions — client_max_body_size defaults to 1m and applies in http, server and location contexts. - ingress-nginx: all versions — Configured through the proxy-body-size annotation rather than nginx.conf. - Platforms: linux ## Not applicable to - 408 Request Timeout, where the body is slow rather than large - 431 Request Header Fields Too Large, which concerns headers and is governed by different directives - Application-level upload limits that return the framework's own error rather than 413 from nginx - Apache's LimitRequestBody, which produces the same status from an unrelated configuration ## Evidence 1. [nginx — ngx_http_core_module, client_max_body_size](https://nginx.org/en/docs/http/ngx_http_core_module.html) — nginx (official-docs), read 2026-08-08 Supports: That client_max_body_size governs the maximum request body, defaults to 1m, and is valid in http, server and location contexts — which is what makes the wrong-context case possible. > Sets the maximum allowed size of the client request body. 2. [413 Content Too Large](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/413) — MDN Web Docs (official-docs), read 2026-08-08 Supports: That 413 means the request entity exceeded a server-defined limit — so any layer in the chain may legitimately emit it, which is why identifying the layer comes before changing configuration. > indicates that the request entity was larger than limits defined by server ## Confidence medium — The directive's meaning, its 1m default and its valid contexts are quoted from nginx's own module reference, and the status semantics from MDN. Confidence is medium rather than high because it rests on two sources: the ingress annotation name and the framework-side limits are drawn from the respective projects' conventions rather than quoted here, and the layer-identification method is diagnostic practice. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"nginx-413-request-entity-too-large","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/nginx-413-request-entity-too-large · knowbase · CC-BY-4.0 --- # nginx 502 Bad Gateway: upstream returned an invalid response nginx reached your application and did not get a usable answer back. The status says nothing about why — that is in nginx's own error log, which names the upstream, the syscall that failed and the reason, and reading it turns guesswork into one lookup. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/http-502-bad-gateway-nginx ## Error signature ``` 502 Bad Gateway ``` Codes: 502 ## Problem Requests return 502 while nginx itself is running fine. The application may be up, reachable directly on its own port, and showing nothing unusual in its logs. Restarting nginx changes nothing because nginx is not what failed — it is reporting that whatever sits behind it did. ## Root cause - **The upstream process is not listening** _(primary)_ - Crashed, still starting, or bound to a different address. Binding to 127.0.0.1 while nginx connects to another interface — or the reverse in a container, where 127.0.0.1 is not shared — produces a refused connection. - How to tell: The error log says connect() failed (111: Connection refused), and ss -ltnp shows nothing listening on the expected address and port - **The upstream took longer than proxy_read_timeout** _(primary)_ - nginx waits 60 seconds by default for a response and then gives up with 502. A slow query or a long export exceeds it while the application is still working normally. - How to tell: The error log says upstream timed out, and the elapsed request time in the access log is close to 60 seconds - **The upstream closed the connection mid-response** _(common)_ - The worker was killed — OOM, a deploy, or a request timeout inside the application — after nginx had already forwarded the request. nginx receives a truncated response rather than none at all. - How to tell: The error log says upstream prematurely closed connection, and the application's own log shows a worker restart or OOM kill at the same timestamp - **Response headers exceed nginx's buffer** _(common)_ - A large Set-Cookie or auth header can overflow proxy_buffer_size, and nginx rejects the response as invalid rather than truncating it. - How to tell: The error log mentions upstream sent too big header, and the failure follows requests carrying unusually large cookies or tokens - **The socket exists but nginx cannot use it** _(common)_ - With a Unix socket, the nginx worker user needs permission on the socket file and every directory above it. SELinux or AppArmor can also deny the connection while permissions look correct. - How to tell: The error log reports permission denied on the socket path, or the denial appears in audit.log rather than in nginx's log - **Every server in the upstream group is marked down** _(edge)_ - After enough failures nginx takes members out of rotation, and once the last one is out it fails immediately without attempting a connection. - How to tell: The error log says no live upstreams, with no connect() attempt recorded ## Solution 1. Read nginx's error log. It names the upstream address, the failing syscall and the errno — which identifies the cause without needing to reproduce anything. ```bash tail -50 /var/log/nginx/error.log | grep -i upstream ``` Note: The distinction between 'Connection refused', 'timed out' and 'prematurely closed' maps directly onto the first three causes; they need different fixes. 2. Check the upstream is listening where nginx is looking. Address mismatches are the single most common cause in containers. ```bash ss -ltnp | grep -E ':(3000|8000|8080)' ; curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3000/ ``` 3. For a refused connection, make the bind address and the proxy_pass target agree. In a container, bind to all interfaces rather than loopback. ```javascript # 🔴 unreachable from another container app.listen(3000, "127.0.0.1"); # ✅ reachable on the container network app.listen(3000, "0.0.0.0"); ``` 4. For a timeout, raise the proxy timeouts only if the request is legitimately slow — otherwise fix the slow path. The default is 60 seconds. ```nginx location /api/export { proxy_pass http://app; proxy_connect_timeout 5s; # reaching the upstream should be fast proxy_read_timeout 300s; # this endpoint is genuinely slow proxy_send_timeout 300s; } ``` Note: Scope generous timeouts to the endpoint that needs them. Raising them globally turns every stuck request into a held worker. 5. For a header-too-big error, give nginx room for the response headers. ```nginx proxy_buffer_size 16k; proxy_buffers 4 16k; proxy_busy_buffers_size 32k; ``` 6. For premature closes, look at why the worker died rather than at nginx. Check the application's memory ceiling and its own request timeout — nginx is the messenger. **Verify:** The request succeeds, and nginx's error log records no new upstream entries during a full traffic cycle including the slowest endpoint. **If that fails:** While the root cause is being fixed, keep the site partially up by serving a cached or static response for the failing location — proxy_cache_use_stale with error and timeout lets nginx answer from cache instead of returning 502. ## Applies to - nginx: all versions — proxy_read_timeout defaults to 60s; proxy_connect_timeout to 60s. - HTTP: RFC 9110 — 502 means a gateway received an invalid response from an upstream server. - Platforms: linux ## Not applicable to - 504 Gateway Timeout, which some proxies return instead of 502 for the timeout case - 503 Service Unavailable, typically emitted deliberately during maintenance or by rate limiting - 500 from the application itself, which reached nginx as a valid response and is passed through - 413 Request Entity Too Large, which nginx rejects before contacting the upstream at all ## Evidence 1. [502 Bad Gateway](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/502) — MDN Web Docs (official-docs), read 2026-08-08 Supports: That 502 means a proxy received an invalid response from upstream — so the failure is always behind the proxy, which is why restarting nginx does not help. > indicates that a server was acting as a gateway or proxy and that it received an invalid response from the upstream server 2. [nginx — ngx_http_proxy_module, proxy_read_timeout](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) — nginx (official-docs), read 2026-08-08 Supports: That the default read timeout is 60 seconds and is configurable per location, which is why slow endpoints fail at a consistent one-minute boundary. > proxy_read_timeout 60s; ## Confidence medium — The meaning of 502 and the 60-second default read timeout are quoted from MDN and nginx's own module reference, and together they explain the two primary causes. Confidence is medium rather than high because it rests on two sources: the specific error-log strings used as discriminators are nginx's runtime messages rather than documented text, and the buffer sizing values are conventional starting points. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"http-502-bad-gateway-nginx","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/http-502-bad-gateway-nginx · knowbase · CC-BY-4.0 --- # Node.js ERR_REQUIRE_ESM: require() of an ES Module is not supported A CommonJS file tried to require() an ES module. Since Node 22.12 this is allowed for modules that are fully synchronous, so on a current runtime the error usually means the module contains top-level await — a different failure that most advice about this error predates. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/node-err-require-esm ## Error signature ``` Error [ERR_REQUIRE_ESM]: require() of ES Module not supported ``` Codes: ERR_REQUIRE_ESM, ERR_REQUIRE_ASYNC_MODULE ## Problem A dependency that worked before now throws on require(), usually after upgrading it to a major version that shipped as ESM-only. The application is CommonJS, the library is not, and the two cannot be mixed by the old rules. Advice found online contradicts itself because the rules changed: what was impossible before Node 22.12 is now permitted for a subset of modules. ## Root cause - **The Node version predates require(esm) support** _(primary)_ - Before Node 22.12 a CommonJS file could not require() an ES module under any circumstances. On those runtimes the error is unconditional and no change to the dependency will fix it. - How to tell: node --version reports below 22.12, and the error code is ERR_REQUIRE_ESM - **The module is not fully synchronous** _(primary)_ - Current Node allows require() of an ES module only if it has no top-level await. An asynchronous module cannot be evaluated synchronously, so the error persists on a new runtime — but with a different code. - How to tell: Node is 22.12 or newer and the code is ERR_REQUIRE_ASYNC_MODULE, or the module source contains await outside any function - **The importing file is being treated as CommonJS unintentionally** _(common)_ - A .js file is CommonJS unless the nearest parent package.json declares "type": "module". Adding an import statement to such a file produces a syntax error rather than an ESM file. - How to tell: The nearest package.json has no type field or says commonjs, while the file uses import syntax - **The dependency published only an ESM entry point** _(common)_ - A library that dropped its CommonJS build has no export the require() condition can resolve. Its major-version notes usually say ESM-only, which is the real change rather than a bug. - How to tell: The package's exports field offers only an import condition, with no require condition or main - **A transpiler emits require() for what is genuinely ESM** _(edge)_ - TypeScript or Babel configured to output CommonJS turns import statements into require() calls, so source that looks like ESM runs as CJS and hits the same wall at runtime. - How to tell: The source uses import but the compiled output in dist contains require(), and module in tsconfig is commonjs ## Solution 1. Establish which of the two errors you have, because the fixes differ. Check the runtime version and the exact error code before changing anything. ```bash node --version && node -e "try{require('the-package')}catch(e){console.log(e.code)}" ``` 2. On Node older than 22.12, upgrade the runtime if you can. This alone resolves the common case of requiring a synchronous ESM-only library. ```bash node --version # target 22.12+ or 24 LTS ``` 3. Where the runtime cannot change, load the module asynchronously. Dynamic import works from CommonJS on every supported version and returns a promise. ```javascript // 🔴 fails when the-package is ESM-only on an older runtime const pkg = require("the-package"); // ✅ works from CommonJS everywhere async function main() { const { default: pkg } = await import("the-package"); return pkg(); } // for a value needed at module scope, keep it lazy let cached; const getPkg = async () => (cached ??= (await import("the-package")).default); ``` Note: The import() call is asynchronous and cannot be made synchronous. If the calling API must stay synchronous, the caller has to change too. 4. If the error is ERR_REQUIRE_ASYNC_MODULE, the module has top-level await and no runtime will require() it. Use dynamic import, or if it is your own module, move the awaited work into an exported async function. ```javascript // 🔴 top-level await makes this module unrequireable const config = await loadConfig(); export { config }; // ✅ defer the await to the caller export const getConfig = async () => loadConfig(); ``` 5. To convert the importing project to ESM instead, declare it and fix the consequences — __dirname, require and .json imports all change. ```json { "type": "module" } ``` Note: Under type module, .js files are ESM and CommonJS-only files must be renamed to .cjs. This is a project-wide change, not a per-file one. 6. If you publish the library, ship both entry points so consumers of either module system resolve correctly. ```json { "exports": { ".": { "require": "./dist/index.cjs", "import": "./dist/index.mjs" } } } ``` **Verify:** The application starts and the module loads without an ERR_REQUIRE_ESM or ERR_REQUIRE_ASYNC_MODULE code, on the same Node version used in production rather than a newer local one. **If that fails:** Where an ESM-only dependency cannot be loaded asynchronously and the runtime cannot move, pin the last version that shipped a CommonJS build and treat the upgrade as scheduled work — an ESM-only major release is a deliberate break, not a defect. ## Applies to - Node.js: 12 and later — require() of a synchronous ES module became supported in 22.12; before that it always failed. Asynchronous modules still cannot be required on any version. - CommonJS: all versions — A .js file is CommonJS unless the nearest package.json sets type to module. - Runtimes: Node.js ## Not applicable to - Cannot use import statement outside a module, which is a parse error in a CJS file rather than a load failure - ERR_MODULE_NOT_FOUND, where the specifier does not resolve at all - Bundled applications where the bundler resolves modules at build time and Node never sees the require - Browser module errors, which have no CommonJS to interoperate with ## Evidence 1. [Node.js — Modules: CommonJS modules, Loading ECMAScript modules using require()](https://nodejs.org/api/modules.html) — OpenJS Foundation (official-docs), read 2026-08-08 Supports: That require() supports ES modules only when they are fully synchronous — no top-level await — which is precisely why the error survives a runtime upgrade for asynchronous modules and disappears for synchronous ones. > require() only supports loading ECMAScript modules that meet the following requirements: ... The module is fully synchronous (contains no top-level await) 2. [Node.js — Modules: Packages, Determining module system](https://nodejs.org/api/packages.html) — OpenJS Foundation (official-docs), read 2026-08-08 Supports: That the module system of a .js file is decided by the nearest parent package.json's type field, which determines whether an import statement is valid in it and whether its requires are CommonJS. > Files with a .js extension when the nearest parent package.json file contains a top-level "type" field with a value of "module" ## Confidence medium — The synchronous-module requirement and the type-field resolution rule are quoted from Node's own API documentation, and together they explain both error codes. Confidence is medium rather than high because it rests on two sources: the version at which require(esm) became available and the dual-export remedy are drawn from release history and ecosystem convention rather than from a sentence I could quote here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"node-err-require-esm","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/node-err-require-esm · knowbase · CC-BY-4.0 --- # Node.js FATAL ERROR: JavaScript heap out of memory V8 hit its heap ceiling and gave up. Two different ceilings can cause it — V8's own --max-old-space-size and the container's memory limit — and they fail differently. Raising the wrong one either does nothing or gets the process killed instead. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/node-javascript-heap-out-of-memory ## Error signature ``` FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ``` Codes: ERR_WORKER_OUT_OF_MEMORY ## Problem A process that works on a developer machine dies in CI or production, or a long-running service crashes after hours of normal operation. The stack trace is V8's rather than the application's, so it names no useful line. Raising the memory limit sometimes helps and sometimes changes nothing, which makes the boundary look arbitrary. ## Root cause - **The workload legitimately needs more heap than V8 allows** _(primary)_ - A large build, a bulk import, or a sizeable JSON parse. V8's old-space limit is a deliberate ceiling rather than a reflection of available RAM, and it is often smaller than the machine's memory. - How to tell: The failure is reproducible at a specific input size, and raising --max-old-space-size moves the boundary rather than merely delaying it - **A genuine leak retains objects across requests** _(primary)_ - A module-level cache without eviction, listeners added per request and never removed, or a closure holding a large buffer. Heap usage grows with uptime rather than with the current request, so any ceiling is reached eventually. - How to tell: Heap after a forced GC grows monotonically over hours, and two heap snapshots taken an hour apart show the same constructor growing - **Data is loaded whole instead of streamed** _(common)_ - Reading a file, an HTTP response, or a query result into memory at once. Memory then scales with the input rather than with concurrency, which works fine until the input grows. - How to tell: Peak memory tracks input size almost exactly, and the code uses readFile, res.json or an unpaginated query rather than a stream or cursor - **The container limit is below the V8 heap setting** _(common)_ - Telling V8 it may use 4 GB inside a 1 GB container means the kernel kills the process before V8 ever reaches its own limit. The symptom then changes from a heap error to exit code 137, which sends you looking in the wrong place. - How to tell: The container reports OOMKilled with exit code 137 rather than printing the V8 fatal error, and --max-old-space-size exceeds the memory limit - **Many small objects rather than a few large ones** _(edge)_ - Millions of small objects, strings or closures can exhaust the heap while no single allocation looks large, and they make garbage collection progressively more expensive before the failure. - How to tell: The log shows repeated ineffective mark-compacts and rising GC pause times before the fatal error ## Solution 1. Determine which ceiling you hit. A printed V8 fatal error means V8's limit; a silent death with exit code 137 means the kernel or container limit. ```bash node -e "console.log(require('v8').getHeapStatistics().heap_size_limit / 1024 / 1024, 'MiB heap limit')" ``` 2. If the workload genuinely needs more heap, raise V8's limit — but keep it below the container's memory limit, with room for everything outside the heap. ```bash # V8 heap ceiling in MiB NODE_OPTIONS=--max-old-space-size=3072 node app.js ``` Note: On a 4 GiB container, 3072 leaves headroom for buffers, native memory and the runtime itself. Setting it at or above the container limit converts a recoverable heap error into a SIGKILL. 3. Make the two limits agree in the container definition, so neither surprises the other. ```yaml resources: limits: memory: "4Gi" env: - name: NODE_OPTIONS value: "--max-old-space-size=3072" ``` 4. If memory grows with uptime, take heap snapshots and compare rather than guessing. The retained-size delta names the leak. ```bash node --heapsnapshot-signal=SIGUSR2 app.js # then: kill -USR2 ``` Note: Take one after warm-up and one an hour later, then load both into Chrome DevTools and compare by constructor. 5. Stream data instead of buffering it whole, so memory scales with concurrency rather than input size. ```javascript // 🔴 whole file in memory const data = await fs.readFile("huge.ndjson", "utf8"); for (const line of data.split("\n")) process(line); // ✅ constant memory regardless of size const rl = readline.createInterface({ input: createReadStream("huge.ndjson") }); for await (const line of rl) process(line); ``` 6. Bound every cache. An unbounded Map at module scope is a leak with extra steps. ```javascript // 🔴 grows forever const cache = new Map(); // ✅ bounded, with eviction import { LRUCache } from "lru-cache"; const cache = new LRUCache({ max: 5000, ttl: 60_000 }); ``` **Verify:** The process completes the workload that previously failed, and over a sustained run heap_used_size after garbage collection returns to a stable baseline rather than climbing. **If that fails:** For a genuinely large one-off job that cannot be streamed, move it out of the request path into a worker or a separate process with its own generous limit, so an out-of-memory failure there cannot take the service down with it. ## Applies to - Node.js: 12 and later — --max-old-space-size sets V8's old-space ceiling in MiB; it is independent of, and can exceed, the memory actually available. - V8: all versions shipped with Node.js - Runtimes: Node.js, containers ## Not applicable to - Exit code 137 with no V8 error printed, which is the kernel OOM killer rather than V8's own limit - Native or off-heap memory growth such as Buffers, which is not governed by the heap limit - Browser tab crashes, where the limit is set by the browser rather than by a flag - Slow performance from garbage-collection pressure without an actual failure ## Evidence 1. [Node.js CLI — --max-old-space-size](https://nodejs.org/api/cli.html) — OpenJS Foundation (official-docs), read 2026-08-08 Supports: That the flag sets V8's old-space ceiling, that approaching it makes V8 spend increasing time in garbage collection before failing, and that the value should be set below available memory to leave room for other uses — which is exactly why it must stay under a container limit. > Sets the max memory size of V8's old memory section. As memory consumption approaches the limit, V8 will spend more time on garbage collection in an effort to free unused memory. 2. [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) — The Kubernetes Authors (official-docs), read 2026-08-08 Supports: That a container exceeding its memory limit is killed by the kernel OOM killer — the reason a V8 heap ceiling set above the container limit produces a SIGKILL instead of a catchable heap error. > memory limits are enforced by the kernel with out of memory (OOM) kills ## Confidence medium — The flag's meaning, V8's escalating garbage collection near the limit, and the guidance to stay below available memory are quoted from Node's own CLI reference; the container interaction is quoted from Kubernetes. Confidence is medium rather than high because it rests on two sources: the snapshot-comparison workflow and the streaming and cache remedies are established practice rather than statements quoted here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"node-javascript-heap-out-of-memory","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/node-javascript-heap-out-of-memory · knowbase · CC-BY-4.0 --- # Npgsql PostgresException 22001: value too long for type character varying(n) PostgreSQL rejects the write because a string exceeds the column's declared length. The error names the type and the limit but not the column, so the practical work is identifying which parameter overflowed — and PostgreSQL counts characters, not bytes. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/npgsql-22001-string-data-right-truncation ## Error signature ``` 22001: value too long for type character varying(n) ``` Codes: 22001, string_data_right_truncation ## Problem An INSERT or UPDATE through Npgsql throws Npgsql.PostgresException with SqlState 22001. The message reports the column type and its length but not which column failed, and with Entity Framework Core batching several rows into one round trip, it is not obvious which entity or property caused it. ## Root cause - **The value is genuinely longer than the column's declared length** _(primary)_ - PostgreSQL enforces varchar(n) as a hard constraint and rejects the statement rather than silently truncating, unlike some other engines running in a permissive mode. - How to tell: The offending .NET string's Length exceeds the n reported in the message - **The EF Core model does not declare a maximum length that matches the schema** _(common)_ - Without HasMaxLength or a MaxLength attribute, EF Core will not validate on the client, so an over-long value travels to the server and fails there. The reverse also happens after schema drift, where the model says 200 and the table still says 50. - How to tell: The property has no MaxLength in the model, or its value disagrees with information_schema.columns - **The column is char(n) rather than varchar(n)** _(common)_ - char(n) is blank-padded to exactly n characters. Trailing whitespace that looks harmless still counts toward the limit and triggers the same error. - How to tell: The message says 'character(n)' rather than 'character varying(n)' - **Character count confused with byte count** _(common)_ - PostgreSQL's n in varchar(n) counts characters, not bytes. Columns sized from a byte budget are wrong in both directions — too small for multi-byte text sized in bytes, and unexpectedly permissive when developers assume the opposite. - How to tell: The string's character length is at or under n but its UTF-8 byte length is larger - **An explicit parameter Size smaller than the value** _(edge)_ - Setting NpgsqlParameter.Size or an explicit NpgsqlDbType with a narrower width can truncate ahead of the server. - How to tell: The failure disappears when the explicit Size is removed from the parameter ## Solution 1. Catch the exception as PostgresException and read SqlState rather than matching on the message text, which is localised and version-dependent. ```csharp try { await db.SaveChangesAsync(); } catch (PostgresException ex) when (ex.SqlState == PostgresErrorCodes.StringDataRightTruncation) { // 22001 — a value exceeded its column's declared length logger.LogError(ex, "Truncation on {Table}", ex.TableName); throw; } ``` Note: PostgresException always populates SqlState; TableName and ColumnName are only present for error classes where the server supplies them, and 22001 generally does not identify the column. 2. Find the offending column by comparing the declared widths against the lengths you are actually sending. ```bash psql -c "SELECT column_name, data_type, character_maximum_length FROM information_schema.columns WHERE table_name = 'your_table' ORDER BY character_maximum_length" ``` 3. Decide whether the schema or the input is wrong. If the limit is a real business rule, validate before the database call so the user gets a field-level message instead of a 500. ```csharp modelBuilder.Entity() .Property(c => c.DisplayName) .HasMaxLength(200) .IsRequired(); ``` 4. If the limit is arbitrary, widen the column. In PostgreSQL, increasing a varchar's length does not rewrite the table and takes only a brief lock. ```bash ALTER TABLE customers ALTER COLUMN display_name TYPE varchar(500); ``` 5. If there is no business limit at all, use text. In PostgreSQL text and varchar share an implementation, so text carries no performance penalty over varchar(n). 6. Size columns in characters, not bytes, and verify with length() versus octet_length() when the data is multi-byte. ```bash SELECT length(val) AS chars, octet_length(val) AS bytes FROM t WHERE id = 1; ``` **Verify:** Re-run the failing write; it should succeed, and a deliberately over-long value should now be rejected by model validation with a field-level error rather than reaching PostgreSQL. **If that fails:** Where the input legitimately exceeds the limit and cannot be rejected — imported third-party data, for example — truncate explicitly at the application boundary and record that truncation, rather than widening a column that encodes a real contract. ## Applies to - PostgreSQL: 9.0 and later — SQLSTATE 22001 is string_data_right_truncation in class 22, Data Exception. - Npgsql: 4.0 and later — PostgresException.SqlState is always present; PostgresErrorCodes exposes named constants. - Entity Framework Core: 3.1 and later — Client-side length validation only happens when the model declares a maximum length. - Runtimes: .NET 6, .NET 8, .NET 9 ## Not applicable to - SQLSTATE 23505 unique_violation, which is a duplicate key rather than a length problem - SQL Server error 8152 or 2628, which is the same class of failure in a different engine with different semantics - Numeric overflow, which raises 22003 numeric_value_out_of_range - PostgreSQL text columns, which have no declared length limit and cannot raise this error ## Evidence 1. [PostgreSQL Error Codes (Appendix A)](https://www.postgresql.org/docs/current/errcodes-appendix.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-07 Supports: That SQLSTATE 22001 is named string_data_right_truncation and belongs to class 22, Data Exception. > 22001 string_data_right_truncation 2. [PostgreSQL Character Types](https://www.postgresql.org/docs/current/datatype-character.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-07 Supports: That varchar(n) rejects over-long values rather than truncating, that char(n) is blank-padded, that n is measured in characters, and that text and varchar perform identically. > There is no performance difference among these three types, apart from increased storage space when using the blank-padded type 3. [Npgsql PostgresException API reference](https://www.npgsql.org/doc/api/Npgsql.PostgresException.html) — Npgsql (official-docs), read 2026-08-07 Supports: That PostgresException exposes SqlState, always populated, alongside TableName, ColumnName, and ConstraintName, which are populated only when the server supplied them. > If the error was associated with a specific table column, the name of the column. 4. [PostgreSQL Error and Notice Message Fields](https://www.postgresql.org/docs/current/protocol-error-fields.html) — The PostgreSQL Global Development Group (specification), read 2026-08-07 Supports: That the column name field is only supplied for specific error classes, which is why 22001 typically arrives without naming the column that overflowed. > The fields for schema name, table name, column name, data type name, and constraint name are supplied only for a limited number of error types ## Confidence high — The error code, the character-versus-byte semantics, and the varchar/text equivalence are stated directly in PostgreSQL's own reference, and the exception surface is taken from Npgsql's API documentation. The claim that 22001 usually omits the column name follows from the protocol's error-field rules rather than from an explicit sentence about 22001, so it is stated as a tendency, not a guarantee. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"npgsql-22001-string-data-right-truncation","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/npgsql-22001-string-data-right-truncation · knowbase · CC-BY-4.0 --- # npm ERESOLVE: unable to resolve dependency tree Two packages demand incompatible versions of the same peer dependency and npm refuses to guess. The two flags everyone reaches for are not equivalent: --legacy-peer-deps ignores peer constraints entirely, while overrides pins one version deliberately. Only the second leaves a tree you can reason about. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/npm-eresolve-peer-dependency ## Error signature ``` npm error code ERESOLVE ``` Codes: ERESOLVE ## Problem An install that worked yesterday fails after one package was upgraded, and CI fails while a developer machine succeeds because the developer has a lockfile and node_modules already. The message prints a tree of conflicting requirements that is hard to read, and the two suggested flags produce very different long-term outcomes with no indication of which is appropriate. ## Root cause - **Two dependencies require incompatible versions of the same peer** _(primary)_ - One package needs react 18, another declares a peer range of react 17 only. Both are internally consistent; together they have no valid solution, and npm reports that rather than choosing. - How to tell: The error tree shows the same package name twice with non-overlapping ranges, one as 'Found' and one as 'Could not resolve' - **A package has not published support for the new major** _(primary)_ - A plugin whose peerDependencies still name the previous major of its host. Frequently it works fine in practice and the metadata is simply behind — but npm can only read the metadata. - How to tell: The blocking package's latest version still declares the old peer range, and its issue tracker has an open request for the new one - **The project is being installed with npm 7 or newer for the first time** _(common)_ - npm 6 ignored peer dependencies; npm 7 onwards enforces them. A tree that installed cleanly for years can fail on a newer npm without anything in the project changing. - How to tell: The same install succeeds with npm 6, and the conflict involves peers that were never previously enforced - **A stale lockfile encodes an arrangement npm will no longer produce** _(common)_ - The lockfile records a resolution from an older algorithm. npm then cannot reconcile what is written with what the current rules allow. - How to tell: Deleting package-lock.json changes the error or resolves it, and the lockfile's lockfileVersion predates the npm in use - **Workspace hoisting produces a conflict the packages do not have individually** _(edge)_ - In a monorepo, two workspaces pinning different majors of a shared peer collide when hoisted to the root even though each workspace is self-consistent. - How to tell: Each workspace installs cleanly on its own, and the conflict appears only from the repository root ## Solution 1. Read the tree properly. 'Found' is what npm intends to install; 'Could not resolve' is the package objecting. Those two lines identify the conflict. ```bash npm install 2>&1 | grep -A20 'ERESOLVE' ``` 2. Check what the objecting package actually requires, rather than inferring it from the error. ```bash npm view peerDependencies ``` 3. Prefer resolving the conflict for real: upgrade the package that lags, or move the shared dependency to a version both accept. This leaves no override to explain later. ```bash npm view versions --json | tail -20 ``` 4. When the metadata is merely behind and you have verified compatibility, pin the version deliberately with overrides. This states an intention, and it applies to the one dependency you decided about rather than to all of them. ```json { "overrides": { "react": "18.3.1" } } ``` Note: Prefer this to --legacy-peer-deps. Overrides are recorded in package.json, apply to a named package, and survive review; a flag in CI is invisible to everyone reading the repository. 5. Understand what --legacy-peer-deps actually does before using it. It does not resolve the conflict — it stops npm considering peer dependencies at all, for every package, so a genuinely incompatible tree installs silently and fails at runtime. ```bash # 🔴 ignores every peer constraint in the tree, not just the conflicting one npm install --legacy-peer-deps # ✅ a scoped, recorded decision # package.json "overrides", then: npm install ``` 6. Regenerate the lockfile if it encodes a stale arrangement, and commit the result so CI and developers resolve identically. ```bash rm -rf node_modules package-lock.json && npm install && git add package-lock.json ``` **Verify:** npm ci completes from a clean checkout with no ERESOLVE and no --legacy-peer-deps, and npm ls shows a single version satisfying every consumer. **If that fails:** If an upstream package is abandoned and blocks an otherwise necessary upgrade, pin it with overrides and record why in the repository — then treat replacing it as scheduled work, because an override is a deferred decision rather than a solved problem. ## Applies to - npm: 7 and later — npm 7 began enforcing peerDependencies; npm 3 through 6 ignored them, which is the behaviour --legacy-peer-deps restores. - Node.js: all versions shipping npm 7+ - Platforms: linux, macos, windows ## Not applicable to - ETARGET, where no published version matches the requested range at all - ENOENT on a missing package.json, which fails before resolution begins - EACCES permission errors during write, which are filesystem rather than dependency problems - Yarn or pnpm resolution failures, which use different algorithms and report differently ## Evidence 1. [npm config — legacy-peer-deps](https://docs.npmjs.com/cli/v11/using-npm/config) — npm, Inc. (official-docs), read 2026-08-08 Supports: That the flag makes npm ignore peerDependencies entirely, as npm 3 through 6 did, and that its use is discouraged precisely because it stops enforcing a contract that meta-dependencies may rely on — which is why it is not equivalent to an override. > Causes npm to completely ignore peerDependencies when building a package tree, as in npm versions 3 through 6. 2. [npm install — configuration, strict-peer-deps](https://docs.npmjs.com/cli/v11/commands/npm-install) — npm, Inc. (official-docs), read 2026-08-08 Supports: That npm resolves deep peer conflicts using the nearest non-peer specification and warns when it does so, and that treating such a warning as fatal is opt-in — which explains why some conflicts warn and others fail outright. > conflicting peerDependencies deep in the dependency graph will be resolved using the nearest non-peer dependency specification ## Confidence medium — What --legacy-peer-deps does, why npm discourages it, and how npm resolves deep peer conflicts by default are quoted from npm's own documentation — and those are the claims the recommendation rests on. Confidence is medium rather than high because it rests on two sources: the npm 7 enforcement boundary and the workspace-hoisting case are drawn from release history and practice rather than quoted here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"npm-eresolve-peer-dependency","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/npm-eresolve-peer-dependency · knowbase · CC-BY-4.0 --- # OAuth error redirect_uri_mismatch: redirect URI is not registered The authorization server compares the redirect_uri you sent against the registered one with simple string comparison — byte for byte. A trailing slash, a different port, or http instead of https makes two URIs that look the same to a human not equal, and the spec requires the server to reject them. > Confidence: high · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/oauth-redirect-uri-mismatch ## Error signature ``` error=redirect_uri_mismatch ``` Codes: redirect_uri_mismatch, invalid_request ## Problem The login flow fails at the authorization server before the user ever reaches the consent screen. The redirect URI in the client configuration looks identical to the one the application sends, yet the server rejects it. It works in one environment and fails in another, which suggests a configuration sync problem when the difference is usually a single character. ## Root cause - **A trailing slash differs between the request and the registration** _(primary)_ - https://app.example.com/callback and https://app.example.com/callback/ are different strings, therefore different URIs. Comparison is exact, so one extra character is a mismatch. - How to tell: Copying the exact value from the error or the request log and diffing it against the registered value reveals a slash at one end only - **Scheme or host differs** _(primary)_ - http versus https, localhost versus 127.0.0.1, www versus apex. Each pair is two distinct URIs under string comparison even though they may reach the same server. - How to tell: The request is over http while the registration is https, or the hostname form differs while resolving to the same address - **The port is present in one and absent in the other** _(common)_ - Registering http://localhost:3000/callback and sending http://localhost/callback — or the reverse — mismatches. A default port is not normalised away before comparison. - How to tell: One side carries an explicit port and the other relies on the protocol default - **The application builds the URI from the incoming request** _(common)_ - Deriving the callback from Host or X-Forwarded-Host means a proxy, a preview deployment or a custom domain silently changes it. The value then depends on how the user arrived rather than on configuration. - How to tell: The redirect_uri in the outgoing request varies between deployments or hostnames rather than being a fixed configured string - **A query string or fragment was appended** _(common)_ - State belongs in the state parameter, not in the redirect URI. Extra query parameters make the URI differ from the registration, and a fragment is not permitted in a redirect URI at all. - How to tell: The sent redirect_uri contains a ? or # that the registered value does not - **The registration is on a different client or environment** _(edge)_ - Staging credentials used against production configuration, or the URI added to a second OAuth client. The client_id and the registration have to belong together. - How to tell: The URI is present in the provider console but under a different client_id than the one the request sends ## Solution 1. Capture the exact string the application sends, rather than the one you believe it sends. Most mismatches are invisible when read and obvious when diffed. ```bash curl -si 'https://your-app.example.com/login' | grep -i '^location:' | tr '&' '\n' | grep redirect_uri ``` 2. Compare the two values byte for byte. Trailing whitespace and percent-encoding differences do not survive a visual check. ```bash diff <(printf '%s' "$SENT_URI") <(printf '%s' "$REGISTERED_URI") && echo identical ``` 3. Register the URI exactly as sent, then stop deriving it. Configure it as a fixed value so it cannot vary with the request. ```javascript // 🔴 depends on how the user reached the app const redirectUri = `${req.protocol}://${req.get("host")}/auth/callback`; // ✅ a configured constant, identical to what is registered const redirectUri = process.env.OAUTH_REDIRECT_URI; // OAUTH_REDIRECT_URI=https://app.example.com/auth/callback ``` Note: Register one URI per environment and select by configuration. Do not attempt to cover several with one entry — there is no wildcard in exact comparison. 4. Put per-request data in the state parameter, which exists for exactly this and is also your CSRF defence. ```javascript const state = crypto.randomUUID(); await store.set(state, { returnTo: "/dashboard" }, { ttl: 600 }); const url = new URL("https://provider.example.com/authorize"); url.searchParams.set("client_id", clientId); url.searchParams.set("redirect_uri", process.env.OAUTH_REDIRECT_URI); url.searchParams.set("state", state); ``` 5. Verify the client_id in the request belongs to the same client where the URI is registered — a URI on the wrong client is invisible to the right one. 6. For preview or ephemeral deployments, route the callback through one stable registered URI and use state to carry the eventual destination, rather than registering a new URI per deployment. **Verify:** The authorization request reaches the consent screen and returns to your callback with a code, and the redirect_uri sent is byte-identical to the registered value in every environment. **If that fails:** Where a provider genuinely requires many callbacks — multi-tenant subdomains, for example — front them with a single registered redirect URI on a dedicated host that receives the code and forwards it internally, so only one URI ever needs registering. ## Applies to - OAuth 2.0: RFC 6749 — The authorization server MUST compare a fully registered redirection URI using simple string comparison; there is no normalisation and no wildcard. - OpenID Connect: Core 1.0 — Inherits the redirect URI matching rules from OAuth 2.0. - Platforms: web, mobile ## Not applicable to - invalid_client, which means the client_id or secret is wrong rather than the callback - access_denied, where the user declined consent and the redirect URI was accepted - invalid_grant on token exchange, which happens after a successful redirect - CORS errors on the token endpoint, which are browser policy rather than URI registration ## Evidence 1. [RFC 6749 — The OAuth 2.0 Authorization Framework, Redirection Endpoint](https://datatracker.ietf.org/doc/html/rfc6749) — IETF (specification), read 2026-08-08 Supports: That the authorization server must compare a fully registered redirection URI to the one in the request using simple string comparison — the normative rule that makes a trailing slash or a scheme difference a rejection rather than a near match. > the authorization server MUST compare the two URIs using simple string comparison 2. [RFC 3986 — URI Generic Syntax, Simple String Comparison](https://datatracker.ietf.org/doc/html/rfc3986) — IETF (specification), read 2026-08-08 Supports: What simple string comparison means: two URIs are equivalent only if identical as character strings, with no normalisation of case, ports, or trailing separators. > If two URIs, when considered as character strings, are identical, then it is safe to conclude that they are equivalent. 3. [Using OAuth 2.0 for Web Server Applications](https://developers.google.com/identity/protocols/oauth2/web-server) — Google (vendor-kb), read 2026-08-08 Supports: That a major provider implements the specification as written — the value must match an authorised redirect URI exactly, and a difference produces this exact error code rather than a warning or a fallback. > match one of the authorized redirect URIs for the OAuth 2.0 client ## Confidence high — Both load-bearing claims are quoted from the normative specifications: RFC 6749 requires simple string comparison, and RFC 3986 defines it as character-for-character identity. Together they account for every primary cause. The provider-specific console behaviour and the multi-tenant fallback are practice rather than specification, and are described as such. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"oauth-redirect-uri-mismatch","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/oauth-redirect-uri-mismatch · knowbase · CC-BY-4.0 --- # PostgreSQL error 40P01: deadlock detected Two transactions each hold a lock the other wants, so PostgreSQL aborts one to break the cycle. Which one it kills is not predictable, so the fix is never in error handling alone — it is acquiring locks in a consistent order, plus retrying the victim. > Confidence: high · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/postgres-40p01-deadlock-detected ## Error signature ``` ERROR: deadlock detected ``` Codes: 40P01, deadlock_detected ## Problem A transaction fails with deadlock detected under concurrent load and succeeds when retried or when run alone. It appears at a rate proportional to traffic, so it is invisible in development and routine in production. The statement named in the error is often not the one at fault — it is simply the one PostgreSQL chose to abort. ## Root cause - **Two code paths lock the same rows in opposite orders** _(primary)_ - One transaction updates A then B, another updates B then A. Each holds what the other needs. This is the textbook case and needs no explicit LOCK statement — ordinary row-level UPDATEs are enough. - How to tell: The DETAIL line names two processes each waiting on the other, and the two statements touch the same tables in reverse order - **A batch operation updates rows in unpredictable order** _(primary)_ - An UPDATE or DELETE over a set of rows locks them in whatever order the plan produces. Two concurrent batches over overlapping sets can interleave and deadlock even though the application code looks sequential. - How to tell: Both statements in the DETAIL are the same bulk statement, differing only in parameters or in the rows matched - **A foreign key forces a lock on the parent row** _(common)_ - Inserting a child row takes a lock on the referenced parent to keep the reference valid. Two transactions inserting children of each other's parents deadlock without either touching the other's table directly. - How to tell: The DETAIL mentions a table the statement does not name, and that table is the target of a foreign key - **Lock upgrade from a read to a write** _(common)_ - A transaction that SELECTs a row and later updates it holds a weaker lock first and needs a stronger one after. Two transactions doing this on the same row block each other symmetrically. - How to tell: Both transactions read the same row before writing it, and adding FOR UPDATE to the initial SELECT removes the deadlock - **Long-running transactions widen the window** _(edge)_ - Deadlock probability scales with how long locks are held. A transaction kept open across an external API call or user input turns a rare interleaving into a frequent one — the cause is duration, not order. - How to tell: Deadlock rate falls sharply when transaction duration is reduced, without any change to lock ordering ## Solution 1. Read the DETAIL and CONTEXT lines in the server log, not just the error. They name both processes, both statements and both locks — the error message alone names only the victim. ```bash grep -A6 'deadlock detected' /var/log/postgresql/postgresql-*.log | tail -40 ``` Note: Set log_lock_waits on so waits that do not yet deadlock are also recorded; they are the early warning for the same lock ordering problem. 2. Establish a single canonical lock order and apply it everywhere. Sorting the keys before touching them is the cheapest way to guarantee it. ```sql -- 🔴 order depends on which transfer arrives first UPDATE accounts SET balance = balance - 100 WHERE id = :from; UPDATE accounts SET balance = balance + 100 WHERE id = :to; -- ✅ always lock the lower id first, so two transfers cannot interleave SELECT id FROM accounts WHERE id IN (:from, :to) ORDER BY id FOR UPDATE; ``` 3. For bulk statements, impose an order explicitly rather than trusting the plan. ```sql UPDATE items SET status = 'done' WHERE id IN (SELECT id FROM items WHERE status = 'queued' ORDER BY id FOR UPDATE); ``` 4. Retry the aborted transaction. Deadlock is a transient, expected condition under concurrency, so 40P01 should be retried rather than surfaced to the user — with backoff and jitter, and a cap. ```javascript // Retry only 40P01 and 40001 — never a constraint violation const RETRYABLE = new Set(["40P01", "40001"]); for (let attempt = 0; attempt < 3; attempt++) { try { return await db.tx(work); } catch (err) { if (!RETRYABLE.has(err.code) || attempt === 2) throw err; await sleep(50 * 2 ** attempt * (0.5 + Math.random())); } } ``` Note: The retry must re-run the whole transaction. Resuming mid-transaction is not possible — the abort rolled everything back. 5. Shorten transactions. Never hold one open across a network call, a queue publish, or anything waiting on a human. 6. Leave deadlock_timeout alone unless you have measured a reason. It is one second by default and controls how long a waiter sits before checking for a cycle — raising it hides real deadlocks for longer rather than preventing them. **Verify:** Under the concurrency that previously produced them, the PostgreSQL log records no new deadlock detected entries over a full traffic cycle, and any that remain are retried transparently rather than reaching the caller. **If that fails:** Where a consistent order genuinely cannot be imposed — independent services touching shared tables — serialise the conflicting operation behind an advisory lock so contention becomes a queue rather than a cycle. ## Applies to - PostgreSQL: 9.0 and later — Detection and automatic abort are server behaviour; deadlock_timeout defaults to 1s. - Platforms: self-hosted, managed PostgreSQL ## Not applicable to - SQLSTATE 40001 serialization_failure, which is an isolation-level conflict rather than a lock cycle - Lock waits that eventually succeed, which are contention rather than deadlock and never abort - MySQL error 1205 lock wait timeout, a timeout on a single lock rather than a detected cycle - Application-level deadlocks between processes, which the database cannot see or resolve ## Evidence 1. [PostgreSQL — Explicit Locking, Deadlocks](https://www.postgresql.org/docs/current/explicit-locking.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-08 Supports: That PostgreSQL detects the cycle and aborts one transaction unpredictably, that row-level locks alone are enough to deadlock without explicit LOCK statements, and that consistent lock ordering plus retrying the victim is the prescribed defence. > The best defense against deadlocks is generally to avoid them by being certain that all applications using a database acquire locks on multiple objects in a consistent order. 2. [PostgreSQL — Lock Management (deadlock_timeout)](https://www.postgresql.org/docs/current/runtime-config-locks.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-08 Supports: That deadlock detection is deferred by deadlock_timeout, one second by default, and that raising it delays reporting of real deadlocks rather than avoiding them. > This is the amount of time to wait on a lock before checking to see if there is a deadlock condition. 3. [PostgreSQL Error Codes (Appendix A)](https://www.postgresql.org/docs/current/errcodes-appendix.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-08 Supports: That 40P01 is deadlock_detected in class 40, Transaction Rollback — the class whose members are by definition safe to retry. > deadlock_detected ## Confidence high — The detection mechanism, the unpredictability of which transaction is aborted, the lock-ordering defence and the deadlock_timeout default are all quoted from PostgreSQL's own documentation. The foreign-key and lock-upgrade cases follow from documented locking behaviour rather than from sentences about deadlocks specifically, and the retry code is illustrative. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"postgres-40p01-deadlock-detected","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/postgres-40p01-deadlock-detected · knowbase · CC-BY-4.0 --- # PostgreSQL error 42P01: relation does not exist The table usually exists. PostgreSQL looked for it along search_path and did not find it there, or the name was folded to lower case and no longer matches a table created with capitals. Both cases report the same error as a genuinely missing table. > Confidence: high · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/postgres-42p01-relation-does-not-exist ## Error signature ``` ERROR: relation "x" does not exist ``` Codes: 42P01, undefined_table ## Problem A query fails saying the relation does not exist, but the table is visibly there in psql or a GUI client. The same SQL works for one user and fails for another, or works in psql and fails from the application, which makes it look like a permissions or connection problem rather than a name resolution one. ## Root cause - **The table is in a schema that is not on search_path** _(primary)_ - Unqualified names are resolved by walking search_path and taking the first match. A table in a schema outside that list is invisible even though it exists in the same database — and the error is identical to it not existing at all. - How to tell: SHOW search_path does not list the schema, while the table appears in information_schema.tables under a different schema name - **The identifier was created quoted with capitals** _(primary)_ - Unquoted names are folded to lower case. A table created as "MyTable" can only be referenced as "MyTable" with quotes; writing MyTable becomes mytable and does not match. ORMs that quote on create but not on query cause exactly this. - How to tell: The table name in information_schema.tables contains capitals, and the query succeeds when the name is double-quoted - **The connection is to the wrong database** _(common)_ - search_path is per-session and schemas are per-database. A connection string pointing at the default database rather than the application's will report every table missing. - How to tell: SELECT current_database(), current_schema() returns something other than what the application expects - **The migration has not run on this environment** _(common)_ - The table exists in development and not in the target. Straightforward, but worth excluding before chasing search_path — it is the only cause where the table is genuinely absent. - How to tell: The table is absent from information_schema.tables in every schema, not merely outside search_path - **The object exists but is not the kind being used** _(edge)_ - Referencing a sequence, view or type where a table is expected, or querying a composite type by name. The name resolves to something, just not to a relation usable in that position. - How to tell: pg_class lists the name with a relkind other than r or p, or the object is a type rather than a relation - **A temporary table from another session** _(edge)_ - Temporary tables live in a per-session schema and vanish with the connection. With a pooled connection the table may have been created on a different physical session than the one now querying. - How to tell: The table was created as TEMPORARY and the application uses a connection pool ## Solution 1. Ask the database where the table actually is, across every schema. This single query separates the four common causes from each other. ```bash psql -c "SELECT table_schema, table_name FROM information_schema.tables WHERE table_name ILIKE '%your_table%';" ``` Note: ILIKE is deliberate — a case-insensitive match reveals a name created with capitals, which an exact match would hide. 2. Check what the failing session can see, since search_path is per-session and may differ from psql's. ```bash psql -c 'SELECT current_database(), current_schema(), current_user;' -c 'SHOW search_path;' ``` 3. If it is a schema problem, prefer qualifying the name over changing the path. A qualified reference resolves the same way for every user and role. ```sql -- ✅ unambiguous regardless of search_path SELECT * FROM analytics.events; -- session-scoped alternative SET search_path TO analytics, public; -- persistent, per role ALTER ROLE app_user SET search_path TO analytics, public; ``` 4. If it is a case problem, decide on one convention and hold it. Lower case unquoted is the path of least resistance in PostgreSQL; the alternative is quoting the name everywhere, forever. ```sql -- created quoted, so it must always be quoted CREATE TABLE "MyTable" (id int); SELECT * FROM MyTable; -- 🔴 folded to mytable, fails SELECT * FROM "MyTable"; -- ✅ -- rename to the convention instead of quoting forever ALTER TABLE "MyTable" RENAME TO my_table; ``` 5. Verify the migration actually ran against this database before assuming name resolution is at fault. ```bash psql -c "SELECT * FROM schema_migrations ORDER BY version DESC LIMIT 5;" ``` 6. For pooled connections, never rely on a temporary table surviving between statements — create it and use it inside one transaction, or use an unlogged table with an explicit lifetime. **Verify:** The failing query runs from the application's own connection, and information_schema.tables confirms the schema and exact spelling the query uses. **If that fails:** Where an ORM generates unqualified names you cannot control, set search_path on the role rather than in application code, so every connection that role opens resolves names the same way regardless of where the query is built. ## Applies to - PostgreSQL: 9.0 and later — Unquoted identifiers fold to lower case, which is incompatible with the SQL standard's upper-case folding — portable code should quote consistently or never. - search_path: all versions — Defaults to "$user", public — so a custom schema is invisible until added. - Platforms: self-hosted, managed PostgreSQL ## Not applicable to - Permission errors (42501), where the relation is found but access is denied - SQLSTATE 42703 undefined_column, which resolves the table but not a column in it - MySQL error 1146 table doesn't exist, which has no search_path equivalent - Connection failures, which never reach name resolution at all ## Evidence 1. [PostgreSQL — The Schema Search Path](https://www.postgresql.org/docs/current/ddl-schemas.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-08 Supports: That unqualified names resolve along search_path, that the first match wins, and that no match produces an error even when the table exists in another schema of the same database — the exact behaviour behind the primary cause. > If there is no match in the search path, an error is reported, even if matching table names exist in other schemas in the database. 2. [PostgreSQL — Identifiers and Key Words](https://www.postgresql.org/docs/current/sql-syntax-lexical.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-08 Supports: That quoting makes an identifier case-sensitive while unquoted names fold to lower case, so a table created as "MyTable" cannot be reached by writing MyTable. > Quoting an identifier also makes it case-sensitive, whereas unquoted names are always folded to lower case. 3. [PostgreSQL Error Codes (Appendix A)](https://www.postgresql.org/docs/current/errcodes-appendix.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-08 Supports: That 42P01 is undefined_table in class 42, Syntax Error or Access Rule Violation — a name-resolution class rather than a missing-object one. > undefined_table ## Confidence high — The search_path resolution rule and the case-folding rule are quoted verbatim from PostgreSQL's own reference, and they account for the two primary causes. The pooled temporary-table case follows from documented temporary-schema behaviour rather than from a sentence about this error, and the diagnostic queries are standard practice rather than documented procedure. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"postgres-42p01-relation-does-not-exist","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/postgres-42p01-relation-does-not-exist · knowbase · CC-BY-4.0 --- # PostgreSQL error 53300: sorry, too many clients already Every connection slot is taken. Raising max_connections is the obvious move and usually the wrong one — each slot costs memory whether it is working or idle. The real question is why so many connections exist, and the answer is almost always pool size multiplied by instance count. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/postgres-53300-too-many-connections ## Error signature ``` FATAL: sorry, too many clients already ``` Codes: 53300, too_many_connections ## Problem New connections are refused while existing ones keep working. The database itself is not slow and CPU looks fine, but every new client — including psql from an operator trying to investigate — is rejected. Restarting the application clears it briefly, then it returns under the same load. ## Root cause - **Pool size multiplied by instance count exceeds max_connections** _(primary)_ - Each application instance opens its own pool. Twenty pods with a pool of ten each is two hundred connections against a default limit of one hundred, and neither number looks alarming on its own. - How to tell: Count rows in pg_stat_activity and compare against instance count times pool size — they match, and most connections are idle rather than active - **Connections are leaked rather than returned** _(common)_ - A code path that takes a connection and never releases it — an early return, a swallowed exception, or a transaction left open. Usage grows monotonically with uptime rather than with load. - How to tell: pg_stat_activity shows connections stuck in 'idle in transaction' with a state_change timestamp minutes or hours old - **Autoscaling multiplied the pools** _(common)_ - Serverless functions and horizontally autoscaled services each open their own connections. Connection count then tracks instance count, not request volume, and a traffic spike exhausts the database before the application is under any real strain. - How to tell: Connection count rises in step with replica or invocation count while queries per second stay flat - **max_connections is genuinely too low for the architecture** _(common)_ - The default is typically 100. That is a starting point, not a sizing decision — but raising it allocates more shared memory and gives every backend its own work_mem allowance, so a large value trades stability for headroom. - How to tell: Connections are legitimately active rather than idle, and the workload genuinely needs concurrency beyond the current limit - **The reserved superuser slots have been consumed** _(edge)_ - Three slots are held back for superusers by default. Once those go too, even an administrator cannot connect to diagnose the problem. - How to tell: Even a superuser connection is refused, and the message mentions slots reserved for non-replication superuser connections ## Solution 1. Find out who is connected and what they are doing before changing any setting. The split between active and idle decides which fix applies. ```bash psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state ORDER BY 2 DESC;" ``` 2. Look for transactions held open. These hold a slot and often a lock, and they are a bug rather than a capacity problem. ```bash psql -c "SELECT pid, usename, application_name, state, now()-state_change AS idle_for, left(query,60) FROM pg_stat_activity WHERE state='idle in transaction' ORDER BY idle_for DESC LIMIT 20;" ``` 3. Do the arithmetic that actually governs this: total connections must fit within max_connections minus the reserved slots. Size the pool per instance from the limit downward, not upward from what feels reasonable. ```text # instances x pool_size + admin headroom <= max_connections # # max_connections 100 # superuser_reserved 3 (default) # admin/monitoring headroom 7 # ------------------------------ # available to apps 90 # 20 instances -> pool_size 4 ``` 4. Put a pooler in front of the database when instance count is the driver. Transaction pooling lets many clients share few server connections, which is the only fix that survives autoscaling. ```ini [databases] appdb = host=127.0.0.1 port=5432 dbname=appdb [pgbouncer] pool_mode = transaction max_client_conn = 1000 default_pool_size = 20 ``` Note: Transaction mode is incompatible with session-level features such as session-scoped prepared statements, advisory locks and LISTEN/NOTIFY — check the driver before switching. 5. Fix leaks at the source rather than papering over them: release connections in a finally block, and set a statement timeout and an idle-in-transaction timeout so a stuck client cannot hold a slot forever. ```sql ALTER ROLE app SET idle_in_transaction_session_timeout = '30s'; ALTER ROLE app SET statement_timeout = '30s'; ``` 6. Only raise max_connections once pooling and leaks are ruled out, and budget the memory first — the parameter can only be changed at server start, so it costs a restart either way. **Verify:** Under peak load, the count from pg_stat_activity stays comfortably below max_connections with most connections active rather than idle, and no client reports 53300 over a full traffic cycle. **If that fails:** To recover a database that is already full, terminate the oldest idle-in-transaction backends to reclaim slots — pg_terminate_backend against the pids found above — then apply the pooling fix before load returns. ## Applies to - PostgreSQL: 9.0 and later — max_connections defaults to about 100 and can only be changed at server start. - PgBouncer: 1.x — Transaction pooling is the mode that decouples client count from server connections. - Platforms: self-hosted, managed PostgreSQL ## Not applicable to - SQLSTATE 53200 out_of_memory, which is a server memory failure rather than slot exhaustion - Client-side pool timeouts such as HikariCP's connection-is-not-available, which occur before the database is reached - Authentication failures (28P01), where the connection is refused for credentials rather than capacity - Managed-service connection limits enforced by the provider's proxy, which report their own error text ## Evidence 1. [PostgreSQL — Connection Settings](https://www.postgresql.org/docs/current/runtime-config-connection.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-07 Supports: That max_connections defaults to roughly 100, can only be set at server start, and that raising it increases shared memory allocation — which is why it is a sizing trade-off rather than a free fix. Also the three reserved superuser slots. > PostgreSQL sizes certain resources based directly on the value of max_connections . Increasing its value leads to higher allocation of those resources, including shared memory. 2. [PostgreSQL Error Codes (Appendix A)](https://www.postgresql.org/docs/current/errcodes-appendix.html) — The PostgreSQL Global Development Group (official-docs), read 2026-08-07 Supports: That SQLSTATE 53300 is named too_many_connections and belongs to class 53, Insufficient Resources. > too_many_connections 3. [PgBouncer features — pooling modes](https://www.pgbouncer.org/features.html) — PgBouncer (official-docs), read 2026-08-07 Supports: That a connection pooler offers session, transaction and statement pooling, and what session pooling implies — a server connection held for the whole client session, which is why it does not solve instance-count multiplication. > When a client connects, a server connection will be assigned to it for the whole duration it stays connected. ## Confidence high — The limit's default, its restart-only nature, its memory cost and the reserved superuser slots are quoted from PostgreSQL's configuration reference, and the pooling modes from PgBouncer's own documentation. The sizing arithmetic and the timeout settings are standard operational practice rather than values any document prescribes, and are presented as a method rather than a rule. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"postgres-53300-too-many-connections","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/postgres-53300-too-many-connections · knowbase · CC-BY-4.0 --- # Python ModuleNotFoundError: No module named 'x' Python looked along sys.path and did not find the module. Which directories are on that path depends on how the interpreter was started — not on where you are standing in the shell — and that is why the same code runs one way and fails another. > Confidence: medium · Verified: 2026-08-08 · Status: fresh · Source: https://knowbase.sh/k/python-modulenotfounderror ## Error signature ``` ModuleNotFoundError: No module named 'x' ``` Codes: ModuleNotFoundError, ImportError ## Problem An import fails even though the package is installed or the file is visibly next to the script. It works in the IDE and fails from the terminal, or works with python script.py and fails with python -m, or works locally and fails in the container. The message names the module but never says where Python actually looked. ## Root cause - **The package is installed into a different interpreter** _(primary)_ - pip and python resolve independently. Installing with a system pip while running a virtual environment's python — or the reverse — puts the package somewhere the running interpreter never looks. - How to tell: python -c "import sys; print(sys.executable)" and which pip point at different installations - **The virtual environment is not active for this process** _(primary)_ - A virtual environment is isolated from the base installation by design, so only what was installed inside it is importable. An IDE, a cron entry, a systemd unit or a Dockerfile CMD that calls python rather than the venv's python gets the base interpreter and none of the project's packages. - How to tell: sys.prefix does not point at the project's .venv directory, or sys.path contains no site-packages under it - **The project root is not on sys.path** _(primary)_ - sys.path starts with the directory of the script being run, not the working directory. Running python src/app.py puts src on the path, so import src.utils fails while import utils works — and running python -m src.app behaves differently again. - How to tell: The import succeeds when run as python -m package.module from the project root but fails as python path/to/module.py - **The package is not installed at all** _(common)_ - The straightforward case, worth excluding early: the requirement is missing from the environment, or the install failed silently and was never checked. - How to tell: pip show reports nothing for the interpreter actually running - **The import name differs from the distribution name** _(common)_ - What you pip install is not always what you import. The distribution and the module it provides are separate names, and the error refers to the module. - How to tell: pip show finds the distribution, but the name after import is not the module it actually installs - **A local file shadows the module** _(edge)_ - A file or directory in the script's own directory with the same name as a library is found first, because that directory comes first on sys.path. Importing it yields your file instead of the library, or fails midway. - How to tell: A file matching the module name exists beside the script, and the module's __file__ points into the project rather than site-packages ## Solution 1. Ask the failing interpreter what it is and where it looks. Every remaining step depends on this answer, and it takes one command. ```bash python -c "import sys; print(sys.executable); print(sys.prefix); [print(' ', p) for p in sys.path]" ``` Note: Run it the exact way the failing code runs — same shell, same venv state, same entry point. Running it differently answers a different question. 2. Confirm the package is installed for that interpreter specifically, rather than for whichever pip happens to be first on PATH. ```bash python -m pip show ``` 3. Install through the interpreter rather than through a bare pip, so the two cannot diverge. ```bash # 🔴 whichever pip is first on PATH pip install requests # ✅ the pip belonging to this interpreter python -m pip install requests # in a container, be explicit about the interpreter too /app/.venv/bin/python -m pip install requests ``` 4. For a project's own modules, install the project instead of manipulating the path. An editable install puts the package on sys.path properly, which fixes every entry point at once — tests, scripts, and the IDE. ```bash # pyproject.toml declares the package; then, once: python -m pip install -e . # now this resolves from any working directory from myproject.utils import helper ``` Note: This is what removes the whole class of problem. sys.path.append in application code is the alternative, and it breaks the moment anything imports differently. 5. Prefer python -m package.module over python path/to/file.py when running project code. The -m form puts the current directory on sys.path; the file form puts the file's own directory there instead. 6. If a name resolves to the wrong thing, check what was actually imported and rename the local file that shadows it. ```python import requests print(requests.__file__) # points into the project => a local file shadows it ``` **Verify:** The import succeeds from the same entry point that failed, and requests.__file__ — or the equivalent for the module in question — resolves under the intended interpreter's site-packages rather than into the project tree. **If that fails:** Where the layout genuinely cannot change — a legacy script tree, for example — set PYTHONPATH for the process rather than editing sys.path in code, so the path is part of how the program is invoked instead of hidden inside it. ## Applies to - Python: 3.6 and later — ModuleNotFoundError is a subclass of ImportError, added in 3.6; earlier versions raise ImportError for the same condition. - venv: 3.3 and later — Virtual environments are isolated from base-environment packages by default. - Platforms: linux, macos, windows ## Not applicable to - ImportError naming a symbol rather than a module, which means the module was found but the attribute was not - Circular imports, where the module is found but is only partly initialised - Compiled-extension failures such as a missing shared library, which report the library rather than the module - Syntax errors inside an imported module, which surface as SyntaxError rather than a missing module ## Evidence 1. [Python tutorial — The Module Search Path](https://docs.python.org/3/tutorial/modules.html) — Python Software Foundation (official-docs), read 2026-08-08 Supports: That imports resolve along sys.path, and that sys.path begins with the directory of the script being run — or the current directory when no file is given — which is the mechanism behind the project-root and shadowing causes. > The directory containing the input script (or the current directory when no file is specified). 2. [Python — venv, Creation of virtual environments](https://docs.python.org/3/library/venv.html) — Python Software Foundation (official-docs), read 2026-08-08 Supports: That a virtual environment is isolated from the base installation by default, so only packages installed inside it are importable — which is why calling the wrong interpreter loses every project dependency at once. > default is isolated from the packages in the base environment ## Confidence medium — The two load-bearing claims — how sys.path is initialised and that virtual environments are isolated by default — are quoted from Python's own documentation. Confidence is medium rather than high because the remaining causes rest on two sources: the pip-versus-interpreter divergence, the distribution-versus-import name distinction and the editable-install remedy are standard packaging practice rather than statements I could quote from a primary source here. ## If this is not your failure Look yours up instead of adapting this one: https://knowbase.sh/search.json?q= To narrow the causes above to the one you have, run the discriminator on each and POST `{"slug":"python-modulenotfounderror","observations":""}` to https://knowbase.sh/diagnose.json. It answers with the cause your observations identify and why the others are excluded. The same three calls as MCP tools: https://knowbase.sh/mcp --- Retrieved from https://knowbase.sh/k/python-modulenotfounderror · knowbase · CC-BY-4.0 --- # React hydration failed: server HTML did not match the client Hydration compares the server-rendered HTML against the first client render and fails when they differ. The cause is almost always one of three things: HTML the browser silently restructured, rendering logic that branches on the environment, or a value that changes between the two renders. > Confidence: high · Verified: 2026-08-07 · Status: fresh · Source: https://knowbase.sh/k/nextjs-react-hydration-mismatch ## Error signature ``` Hydration failed because the server rendered HTML didn't match the client ``` Codes: 418, 423, 425 ## Problem The page renders on the server, reaches the browser, and then React throws during hydration. The UI may flash correct content and then break, lose event handlers, or re-render entirely on the client. The error frequently appears only in production, or only for some users, which makes it look intermittent when it is not. ## Root cause - **Invalid HTML nesting that the browser restructures before React sees it** _(primary)_ - A
inside a

, nested

tags, a