beginner35 minby DevFox

Debugging a Pod That Will Not Run

A method for Pending, ImagePullBackOff, CrashLoopBackOff and the pod that is Running but wrong — describe, events, logs, previous logs, exec, and what each one tells you.

Kubernetes

Every previous lesson mentioned a symptom in passing — Pending, ImagePullBackOff, CrashLoopBackOff, READY 0/1, OOMKilled. This is the lesson that collects them into one order of operations, so that a pod which will not run stops being a mystery and becomes a short sequence of questions.

By the end of it you will have a method you can apply to any failing pod, and a mapping from every status you are likely to see to the thing that causes it.

The method

Four commands, in this order, on the pod that is wrong. The discipline is doing them in order rather than jumping to logs.

kubectl get pods — what is the STATUS?
And the READY column, which is a separate piece of information people routinely skip past.
kubectl describe pod — what does the cluster say?
The Events list at the bottom is the single most valuable thing in this lesson. Read it as a timeline.
kubectl logs --previous — what does the app say?
--previous reads the attempt that already crashed, which is the one with the error in it.
kubectl exec — what does it look like inside?
Only possible while it is running. For a crash-looping pod, run a copy with its command replaced by a shell.

Half of all pod failures never start a container, so there are no logs to read at all

bash

kubectl describe pod api-6c9f7d4b58-mn4pq

The bottom of that output is the events list, and it is the single most valuable thing in this lesson:

text

Events:
  Type     Reason     Age   From      Message
  ----     ------     ----  ----      -------
  Normal   Scheduled  2m    default-scheduler  Assigned to worker2
  Normal   Pulling    2m    kubelet   Pulling image "notes-api:0.2.0"
  Warning  Failed     2m    kubelet   Failed to pull image: not found
  Warning  Failed     2m    kubelet   Error: ErrImagePull
  Normal   BackOff    89s   kubelet   Back-off pulling image

Read it as a timeline. That one says the scheduler placed the pod successfully, so scheduling is fine, and the kubelet could not pull the image — which is a registry or tag problem, not an application problem.

Pending: it has not been placed

Pending means the pod exists and no node is running it. Almost always the scheduler could not place it, and it says why:

text

Warning  FailedScheduling  0/3 nodes are available: 1 node(s) had
         untolerated taint, 2 Insufficient memory.

Message	Fix
`Insufficient cpu` / `memory`	Lower the request, or add a node
`untolerated taint`	The node is reserved; add a toleration
`didn't match node selector`	Your `nodeSelector` matches nothing
`didn't find available persistent volumes`	The PVC cannot be bound
`too many pods`	The node's pod cap is reached

bash

kubectl describe pod <name> | tail -20     # the scheduler's reasons
kubectl describe node <name>               # requests vs allocatable
kubectl get pvc                            # is a claim also Pending

Remember the arithmetic: the scheduler counts requests, not usage. A cluster of idle machines can be full.

ImagePullBackOff: the image did not arrive

The pod is on a node and the kubelet cannot fetch its image. Four causes, and the event text distinguishes them:

text

Failed to pull image "notes-api:0.2.0": not found
    -> the tag does not exist. Check it against the registry.

Failed to pull image "ghcr.io/me/api:1.0": unauthorized
    -> a private registry with no credentials.
       Create a docker-registry Secret and reference it in
       imagePullSecrets.

Failed to pull image ...: dial tcp: i/o timeout
    -> the node cannot reach the registry. Network or firewall.

exec format error   (once it is running)
    -> wrong architecture: an arm64 image on an amd64 node.
       Build with --platform linux/amd64, or multi-platform.

bash

kubectl describe pod <name> | grep -A5 Events
kubectl get pod <name> -o jsonpath='{.spec.containers[*].image}'

That last command is worth having: it prints the image the pod actually asked for, which is occasionally not the one you believe you deployed.

One local-cluster wrinkle: an image you built on your machine does not exist inside a kind or minikube cluster. It has to be loaded in.

bash

kind load docker-image notes-api:0.2.0 --name learning
minikube image load notes-api:0.2.0

CrashLoopBackOff: it starts and dies

The container runs, exits, and is restarted — repeatedly, with an increasing delay. The status is a symptom; the cause is in the logs of the attempt that already ended:

bash

kubectl logs <name> --previous

--previous is the crucial flag. Plain kubectl logs reads the current attempt, which may be seconds old or not started yet. --previous reads the one that crashed.

bash

kubectl logs <name> --previous --tail=50
kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState}'

What you will find there:

In the logs	Cause
`KeyError: 'DATABASE_URL'`	Missing ConfigMap or Secret key
`could not connect to server`	Dependency unreachable or not ready
`Permission denied`	Non-root user, unwritable mount
`exec ... no such file`	Bad command, or CRLF line endings
Nothing at all, exit 0	The process finished — a daemonising server
`Reason: OOMKilled`	Memory limit too low, or a leak

The empty-log case is the one that puzzles people, and it is the same issue as in Docker: a container lives exactly as long as its main process, so a server that daemonises exits immediately with code 0.

bash

kubectl describe pod <name> | grep -A4 'Last State'

That block names OOMKilled explicitly, which saves a long detour.

Running but READY 0/1

This is the case people misread most often, because the status column says Running and everything looks fine.

text

NAME                   READY   STATUS    RESTARTS   AGE
api-6c9f7d4b58-mn4pq   0/1     Running   0          3m

READY 0/1 means the container is up and failing its readiness probe. It is therefore not in any Service's endpoints and receives no traffic — which is why your Service returns nothing while the pods look healthy.

bash

kubectl describe pod <name> | grep -A3 Unhealthy
kubectl exec <name> -- curl -sv localhost:8000/ready

The second command settles it, and the answer splits the problem cleanly in two.

The endpoint answers from inside

The probe is misconfigured.

Wrong port, wrong path, or timeoutSeconds too tight for an endpoint that touches a database.

Your application is fine. Fix the YAML.

It does not answer

The probe is right.

The application genuinely is not ready, and the probe is doing exactly its job.

Read the logs.

Same symptom, opposite fixes. One command tells you which.

Rising RESTARTS with no crash in the logs is the same story for liveness: the probe is killing a container that was working.

The pod is fine and nothing can reach it

When the pods are healthy, the problem is in the layer above, and there is one command that resolves most of it:

bash

kubectl describe service api | grep Endpoints

Empty endpoints has exactly two causes: no pod matches the selector, or no matching pod is ready.

bash

kubectl get pods -l app=api --show-labels    # do the labels match
kubectl get pods -l app=api                  # are any of them READY

--show-labels is the honest check: it prints what the labels are, rather than what you intended to set.

Then work outwards from inside the cluster:

bash

kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- sh
#   nslookup api            does the Service name resolve
#   wget -qO- http://api    does anything answer

If that works and outside traffic does not, the problem is the Ingress and the previous lesson's table applies — 404 means no rule matched, 503 means a rule matched an empty backend.

The mistake: reading the wrong pod's logs

Bad — reads whichever pod the selector happened to pick first:

bash

kubectl logs -l app=api --tail=20

Good — finds the unhealthy pod, then reads that one:

bash

kubectl get pods -l app=api          # find the one that is not READY
kubectl logs api-6c9f7d4b58-mn4pq --previous

With three replicas and one broken, a label-selected log read returns output from all of them interleaved, or truncates to a subset — and crucially it reads current logs, not the crashed attempt. So you see two healthy pods serving happily and conclude the application is fine, while the failing pod's actual error is in a log you never opened.

kubectl logs -l is excellent for a question about the application as a whole. For a question about one broken pod, name the pod.

Tools worth having

Three that repay the setup time:

bash

kubectl get events --sort-by=.lastTimestamp        # namespace timeline
kubectl get events -A --field-selector type=Warning
kubectl debug -it <pod> --image=busybox:1.36 --target=api

kubectl debug attaches an ephemeral container to a running pod, sharing its namespaces. That is the answer to debugging a distroless image with no shell in it: you bring your own tools alongside the container instead of into it.

And k9s, a terminal UI over the cluster, which turns the four-command method into moving around a list. It is not required and it is noticeably faster once you are in the habit.

Cheat sheet

bash

# The method, in this order
kubectl get pods                        # 1. STATUS and READY
kubectl describe pod <name>             # 2. Events, at the bottom
kubectl logs <name> --previous          # 3. the attempt that crashed
kubectl exec -it <name> -- sh           # 4. inside, if it is running

# Status -> cause
#   Pending             not scheduled: describe gives the reason
#   ContainerCreating   pulling the image, or mounting a volume
#   ImagePullBackOff    bad tag / no credentials / unreachable registry
#   CrashLoopBackOff    starts and exits: logs --previous
#   Running, READY 0/1  failing its readiness probe
#   OOMKilled           over the memory limit (exit 137)
#   Completed           the process finished — often a daemonising server
#   Terminating (stuck) a finalizer, or an unresponsive SIGTERM handler

# Pending
kubectl describe pod <name> | tail -20  # the scheduler's per-node reasons
kubectl describe node <name>            # requests vs allocatable
kubectl get pvc                         # a claim that cannot bind

# Image problems
kubectl get pod <name> -o jsonpath='{.spec.containers[*].image}'
kind load docker-image notes-api:0.2.0 --name learning   # local images
minikube image load notes-api:0.2.0

# Crashes
kubectl logs <name> --previous --tail=50
kubectl describe pod <name> | grep -A4 'Last State'      # names OOMKilled
kubectl run debug --rm -it --image=notes-api:0.2.0 --command -- sh

# Probes
kubectl describe pod <name> | grep -A3 Unhealthy
kubectl exec <name> -- curl -sv localhost:8000/ready

# Nothing can reach a healthy pod
kubectl describe service api | grep Endpoints    # empty = the answer
kubectl get pods -l app=api --show-labels        # do the labels match
kubectl run tmp --rm -it --image=busybox:1.36 --restart=Never -- sh
#   nslookup api ; wget -qO- http://api

# Wider view
kubectl get events --sort-by=.lastTimestamp
kubectl get events -A --field-selector type=Warning
kubectl debug -it <pod> --image=busybox:1.36 --target=api  # no shell?
kubectl get pod <name> -o yaml                   # the whole truth

Where to go next

The last lesson is the capstone: the notes application from the Docker course, deployed to your cluster with everything from these fifteen lessons — a Deployment, a Service, an Ingress, configuration, storage, probes, resource requests — then updated and rolled back.

Before that, cause a couple of these failures deliberately. Deploy an image tag that does not exist; set a memory limit of 10Mi; point a readiness probe at a path your application does not serve. Recognising each status on sight is worth more than reading the table twice.