beginner30 minby DevFox

Probes, Restarts and Self-Healing

Liveness, readiness and startup probes: how the cluster decides your app is alive, ready for traffic, or beyond saving — and what a badly written probe costs.

Kubernetes

A process that has crashed is easy to handle: it exits, the kubelet restarts it. The hard case is the process that is still running and no longer working — deadlocked, out of database connections, stuck on a request that will never complete. Its port is open, kubectl get pods says Running, and every request routed to it fails.

Probes are how you tell the cluster the difference. By the end of this lesson you will know the three kinds, which one removes a pod from traffic and which one kills it, and why a badly written probe is considerably worse than none at all.

Running is not the same as working

Without a probe, "healthy" means one thing: the container's main process has not exited. That is a low bar.

An application can hold its port open while its connection pool is exhausted, its event loop is blocked, a dependency it needs is unreachable, or it is still loading a 4 GB model into memory. In every case the process is alive and the answers are wrong or absent.

Kubernetes cannot infer any of this. Only your application knows whether it can do its job, so it has to be asked — and probes are the asking.

Three probes, three jobs

The names are similar and the consequences are not, which is why getting them straight matters more than the syntax.

readiness

Can it serve traffic now?

Failing drops it from the Service endpoints and nothing else. The pod keeps running, and traffic returns when it passes again.

Reversible, gentle, and where dependency checks belong.

liveness

Is it beyond saving?

Failing kills the container. Use it only for states a restart genuinely fixes.

startup

Has it finished starting?

While it is failing, the other two are not evaluated at all — so a long initialisation cannot be mistaken for a hang.

The syntax is nearly identical. The consequence of a failure is not.

Writing them

Three mechanisms are available for any probe: an HTTP request, a TCP connection, or a command run inside the container.

Any HTTP status from 200 to 399 is a pass. The port can be named, which is worth doing — port: http follows the container port wherever it moves.

The timing fields, and sensible starting values:

Field	Default	Meaning
`initialDelaySeconds`	0	Wait before the first check
`periodSeconds`	10	How often
`timeoutSeconds`	1	How long to wait for an answer
`failureThreshold`	3	Consecutive failures before acting
`successThreshold`	1	Consecutive passes to recover

timeoutSeconds: 1 is the default most worth changing. One second is tight for an endpoint that touches a database, and a probe that times out counts as a failure — which for a liveness probe means a restart caused by nothing but a slow answer.

What the endpoints should check

The two probes should not hit the same endpoint, because they are asking different questions. This is the whole art of it.

Readiness should check dependencies. Can I reach the database? Is my cache available? Is my configuration loaded? If any answer is no, there is no point sending me requests — but do not restart me, because the problem is not mine.

Liveness should check only itself. Is my event loop turning? Am I past a deadlock? A liveness endpoint should not touch a database, because a database outage would then restart every pod you have — which is the mistake below.

The liveness probe that checks a database

Bad — liveness depends on something outside the container:

Good — liveness checks the process, readiness checks the dependencies:

The bad version turns a thirty-second database blip into an outage, and it does so in five steps.

The database is briefly unavailable
Thirty seconds. Nothing is wrong with your application.
Every pod fails the probe at once
They all check the same database, so they all fail simultaneously. This is not a partial outage any more.
Every container is killed and restarted
Because it was a liveness probe. Restarting was never going to help — the problem was never in the container.
They restart into the same unavailable database
Fail again, get killed again. Now the whole Deployment is in CrashLoopBackOff.
Backoff extends the outage past the cause
Ten seconds, twenty, forty, up to five minutes. The database came back three minutes ago.

The database recovers and your application does not

Slow starters

An application that takes ninety seconds to start presents a real dilemma with only two probes. A liveness probe with a short period kills it before it finishes; one lenient enough to allow ninety seconds is also ninety seconds slow to notice a genuine hang later.

initialDelaySeconds is the old workaround, and it is a bad one — you are guessing a fixed number, and a slow day breaks it.

The startup probe removes the trade-off:

Until the startup probe passes, liveness and readiness are not evaluated. The moment it passes, it is never run again and the strict liveness probe takes over. Generous at the start, strict afterwards.

Shutting down without dropping requests

Probes handle the start. The end has its own race, and it is worth understanding because it is the last source of errors during an otherwise clean rollout.

When a pod is deleted, two things happen in parallel: it is removed from the Service endpoints, and its container receives SIGTERM. Those are not synchronised. Endpoint removal has to propagate to kube-proxy on every node, and for a moment traffic still arrives at a pod that has already begun shutting down.

The standard fix is a small deliberate pause before shutting down:

Five seconds is enough for endpoint removal to propagate; the application then handles SIGTERM, finishes its in-flight requests, and exits — all inside the thirty-second grace period, after which SIGKILL arrives regardless.

That only works if your process actually receives SIGTERM, which means the bracket-form CMD from the Docker course. A shell between Kubernetes and your program swallows the signal, and the pod is killed hard thirty seconds later.

Reading a failing probe

Probe failures appear in the pod's events, in the cluster's own words:

Three things to check in order when a probe misbehaves:

That last command is the decisive one. If the endpoint answers from inside the container, the probe's configuration is wrong — wrong port, wrong path, timeout too short. If it does not answer, the probe is correct and your application is genuinely not ready.

A pod with rising RESTARTS and no crash in its logs is a liveness probe doing its job on an application that is not actually broken. kubectl logs --previous shows what the killed container was saying.

Cheat sheet

Where to go next

The cluster can now tell working from running, which is the last piece the scheduler needs before it can make good decisions — and those decisions depend on knowing how much CPU and memory each pod wants. That is the next lesson, along with why a pod sits Pending forever and why another keeps getting OOMKilled.

Before that, add a readiness probe to something you are running and watch kubectl get pods during a rollout. Seeing READY 0/1 for a few seconds before a new pod takes traffic is the mechanism working.