beginner35 minby DevFox

Debugging a Pod That Will Not Run

A method for Pending, ImagePullBackOff, CrashLoopBackOff and the pod that is Running but wrong — describe, events, logs, previous logs, exec, and what each one tells you.

Kubernetes

Every previous lesson mentioned a symptom in passing — Pending, ImagePullBackOff, CrashLoopBackOff, READY 0/1, OOMKilled. This is the lesson that collects them into one order of operations, so that a pod which will not run stops being a mystery and becomes a short sequence of questions.

By the end of it you will have a method you can apply to any failing pod, and a mapping from every status you are likely to see to the thing that causes it.

The method

Four commands, in this order, on the pod that is wrong. The discipline is doing them in order rather than jumping to logs.

kubectl get pods — what is the STATUS?
And the READY column, which is a separate piece of information people routinely skip past.
kubectl describe pod — what does the cluster say?
The Events list at the bottom is the single most valuable thing in this lesson. Read it as a timeline.
kubectl logs --previous — what does the app say?
--previous reads the attempt that already crashed, which is the one with the error in it.
kubectl exec — what does it look like inside?
Only possible while it is running. For a crash-looping pod, run a copy with its command replaced by a shell.

Half of all pod failures never start a container, so there are no logs to read at all

bash

kubectl describe pod api-6c9f7d4b58-mn4pq

The bottom of that output is the events list, and it is the single most valuable thing in this lesson:

text

Events:
  Type     Reason     Age   From      Message
  ----     ------     ----  ----      -------
  Normal   Scheduled  2m    default-scheduler  Assigned to worker2
  Normal   Pulling    2m    kubelet   Pulling image "notes-api:0.2.0"
  Warning  Failed     2m    kubelet   Failed to pull image: not found
  Warning  Failed     2m    kubelet   Error: ErrImagePull
  Normal   BackOff    89s   kubelet   Back-off pulling image

Read it as a timeline. That one says the scheduler placed the pod successfully, so scheduling is fine, and the kubelet could not pull the image — which is a registry or tag problem, not an application problem.

Pending: it has not been placed

Pending means the pod exists and no node is running it. Almost always the scheduler could not place it, and it says why:

Message	Fix
`Insufficient cpu` / `memory`	Lower the request, or add a node
`untolerated taint`	The node is reserved; add a toleration
`didn't match node selector`	Your `nodeSelector` matches nothing
`didn't find available persistent volumes`	The PVC cannot be bound
`too many pods`	The node's pod cap is reached

Remember the arithmetic: the scheduler counts requests, not usage. A cluster of idle machines can be full.

ImagePullBackOff: the image did not arrive

The pod is on a node and the kubelet cannot fetch its image. Four causes, and the event text distinguishes them:

That last command is worth having: it prints the image the pod actually asked for, which is occasionally not the one you believe you deployed.

One local-cluster wrinkle: an image you built on your machine does not exist inside a kind or minikube cluster. It has to be loaded in.

CrashLoopBackOff: it starts and dies

The container runs, exits, and is restarted — repeatedly, with an increasing delay. The status is a symptom; the cause is in the logs of the attempt that already ended:

--previous is the crucial flag. Plain kubectl logs reads the current attempt, which may be seconds old or not started yet. --previous reads the one that crashed.

What you will find there:

In the logs	Cause
`KeyError: 'DATABASE_URL'`	Missing ConfigMap or Secret key
`could not connect to server`	Dependency unreachable or not ready
`Permission denied`	Non-root user, unwritable mount
`exec ... no such file`	Bad command, or CRLF line endings
Nothing at all, exit 0	The process finished — a daemonising server
`Reason: OOMKilled`	Memory limit too low, or a leak

The empty-log case is the one that puzzles people, and it is the same issue as in Docker: a container lives exactly as long as its main process, so a server that daemonises exits immediately with code 0.

That block names OOMKilled explicitly, which saves a long detour.

Running but READY 0/1

This is the case people misread most often, because the status column says Running and everything looks fine.

READY 0/1 means the container is up and failing its readiness probe. It is therefore not in any Service's endpoints and receives no traffic — which is why your Service returns nothing while the pods look healthy.

The second command settles it, and the answer splits the problem cleanly in two.

The endpoint answers from inside

The probe is misconfigured.

Wrong port, wrong path, or timeoutSeconds too tight for an endpoint that touches a database.

Your application is fine. Fix the YAML.

It does not answer

The probe is right.

The application genuinely is not ready, and the probe is doing exactly its job.

Read the logs.

Same symptom, opposite fixes. One command tells you which.

Rising RESTARTS with no crash in the logs is the same story for liveness: the probe is killing a container that was working.

The pod is fine and nothing can reach it

When the pods are healthy, the problem is in the layer above, and there is one command that resolves most of it:

Empty endpoints has exactly two causes: no pod matches the selector, or no matching pod is ready.

--show-labels is the honest check: it prints what the labels are, rather than what you intended to set.

Then work outwards from inside the cluster:

If that works and outside traffic does not, the problem is the Ingress and the previous lesson's table applies — 404 means no rule matched, 503 means a rule matched an empty backend.

The mistake: reading the wrong pod's logs

Bad — reads whichever pod the selector happened to pick first:

Good — finds the unhealthy pod, then reads that one:

With three replicas and one broken, a label-selected log read returns output from all of them interleaved, or truncates to a subset — and crucially it reads current logs, not the crashed attempt. So you see two healthy pods serving happily and conclude the application is fine, while the failing pod's actual error is in a log you never opened.

kubectl logs -l is excellent for a question about the application as a whole. For a question about one broken pod, name the pod.

Tools worth having

Three that repay the setup time:

kubectl debug attaches an ephemeral container to a running pod, sharing its namespaces. That is the answer to debugging a distroless image with no shell in it: you bring your own tools alongside the container instead of into it.

And k9s, a terminal UI over the cluster, which turns the four-command method into moving around a list. It is not required and it is noticeably faster once you are in the habit.

Cheat sheet

Where to go next

The last lesson is the capstone: the notes application from the Docker course, deployed to your cluster with everything from these fifteen lessons — a Deployment, a Service, an Ingress, configuration, storage, probes, resource requests — then updated and rolled back.

Before that, cause a couple of these failures deliberately. Deploy an image tag that does not exist; set a memory limit of 10Mi; point a readiness probe at a path your application does not serve. Recognising each status on sight is worth more than reading the table twice.