beginner30 minby DevFox

Rolling Updates and Rollbacks

Shipping a new version without dropping a request: rollout strategy, maxSurge and maxUnavailable, watching a rollout, and undoing one that went wrong.

Kubernetes

Deploying with Compose meant stopping the old container and starting the new one, and accepting that requests failed in between. That was the fourth thing on the list of problems Kubernetes exists to absorb, and this is the lesson where it gets absorbed.

By the end of it you will have run a rolling update and watched it happen pod by pod, you will know what maxSurge and maxUnavailable control, and you will be able to stop or reverse a bad release in one command.

What triggers a rollout

A rollout starts when the pod template changes — anything under spec.template. A new image, an added environment variable, a modified probe, a new annotation.

Changing something outside the template does not: replicas is a scaling operation, not a rollout, because the existing pods are still correct.

bash

# these start a rollout
kubectl set image deployment/api api=notes-api:0.2.0
kubectl apply -f deployment.yaml        # if the template changed
kubectl rollout restart deployment/api  # a template annotation

# this does not
kubectl scale deployment api --replicas=5

kubectl rollout restart is worth knowing precisely: it stamps a timestamp annotation into the template, which is a change, which starts a normal rolling update. That is how you restart every pod without downtime — and the correct response to a changed ConfigMap that your pods read as environment variables.

Two ReplicaSets, passing each other

The mechanism is the one the Deployments lesson hinted at. A Deployment does not modify pods; it creates a new ReplicaSet for the new template and moves replicas from the old one to the new one, a few at a time.

bash

kubectl set image deployment/api api=notes-api:0.2.0
kubectl get replicasets --watch

text

NAME             DESIRED   CURRENT   READY   AGE
api-6c9f7d4b58   3         3         3       10m    # old
api-7d8b5f9c64   1         1         0       2s     # new
api-6c9f7d4b58   3         3         3       10m
api-7d8b5f9c64   1         1         1       8s     # first one ready
api-6c9f7d4b58   2         2         2       10m    # one old removed
api-7d8b5f9c64   2         2         1       9s
...
api-6c9f7d4b58   0         0         0       11m    # old drained
api-7d8b5f9c64   3         3         3       25s    # new at full

Read the order carefully, because it is the whole guarantee.

A new pod is created
On the new ReplicaSet. The old three are untouched and still serving.
Wait for it to become ready
This is the readiness probe from the last lesson, and it is the entire safety mechanism. Nothing proceeds until it passes.
Only then, remove one old pod
It leaves the endpoints first, then shuts down inside its grace period.

Repeat until the old ReplicaSet is at zero. Capacity never drops below what you asked for.

That guarantee rests entirely on the readiness probe from the last lesson. Without one, a pod counts as ready the moment its process starts, and the rollout cheerfully replaces every working pod with pods that are not yet able to serve — a zero-downtime deployment that drops every request.

Watching and controlling one

rollout status blocks until the rollout settles and exits non-zero if it fails, which makes it the right last line of a deploy script — the pipeline waits for the release rather than declaring success on apply.

Two commands for a rollout you are unsure about:

Pausing is useful mid-release: you now have some pods on the new version and some on the old, and you can look at logs and metrics before committing. It is also the crude version of a canary — one new pod serving a fraction of traffic while you watch.

maxSurge and maxUnavailable

Two fields decide how aggressive the replacement is:

Those are the defaults. With ten replicas, 25% each way means the rollout may run up to 12 pods at once and must keep at least 8 available. Both accept a percentage or an absolute number.

Three configurations worth recognising.

maxSurge: 1, maxUnavailable: 0

Full capacity throughout. The safest.

One extra pod exists at a time, and no old pod leaves until a new one is ready.

Also the slowest — and it cannot progress if the cluster has no room for one more pod, which presents as a rollout that simply stops with the new pod Pending.

maxSurge: 50%, maxUnavailable: 50%

Fast, and capacity dips.

Half the fleet is replaced at once. Fine when you have headroom and the release is routine.

maxSurge: 0, maxUnavailable: 1

Never exceeds replicas.

For a cluster with no spare room, or a workload whose licence or connection limit is tied to the replica count. Capacity dips by one throughout.

You are trading speed against capacity. There is no setting that gives you both.

There is also the other strategy, and its one legitimate use:

Recreate kills every old pod before creating any new ones — deliberate downtime. It is correct when two versions genuinely cannot coexist: a schema migration that the old code cannot read, or a single writer holding an exclusive lock on a volume.

Rolling back

Kubernetes keeps the old ReplicaSets, scaled to zero, which is what makes rollback cheap:

A rollback is a rolling update in reverse — the old ReplicaSet scales up, the new one scales down, with the same readiness guarantees. It is the one command worth having in muscle memory before you need it.

Those <none> entries are a missed opportunity. Record what each revision was:

How much history you keep is configurable, and ten is the default:

The mistake: a tag that does not change

Bad — the image tag is the same, so nothing rolls out:

Good — a new tag makes the template different, so a rollout happens:

A rollout is triggered by a change to the template, and image: myname/notes-api:latest is character-for-character identical before and after your push. Kubernetes has nothing to react to, reports unchanged, and the old pods keep running the old code — while every artefact you have says the deploy succeeded.

Worse, if a pod is recreated later for an unrelated reason, it pulls whatever latest points at then. Two pods of the same Deployment end up running different code with no record of it.

The fix is a unique tag per build — a version, or the commit SHA — which also means the Deployment manifest states exactly what is deployed.

When a rollout gets stuck

A rollout that never finishes has a short list of causes:

Symptom	Cause
New pod `ImagePullBackOff`	Bad tag, or missing pull secret
New pod `Running`, `READY 0/1`	Failing its readiness probe
New pod `CrashLoopBackOff`	Broken configuration or code
New pod `Pending`	No room for the surge pod
`ProgressDeadlineExceeded`	It has not progressed for 600s

The good news is where you are while this happens: a stuck rollout has made no progress, so the old pods are still serving. maxUnavailable: 0 means capacity was never reduced. You are not in an outage — you are in a paused release, and the correct move is usually:

Diagnose afterwards, with production restored.

progressDeadlineSeconds is what produces that last condition, and 600 is the default:

It marks the Deployment as failed after ten minutes of no progress. Note that it does not roll back automatically — Kubernetes has no automatic rollback. Tools built on top of it do: Argo Rollouts and Flagger run canary and blue-green releases with metric-based automatic reversion.

Cheat sheet

Where to go next

You can ship and unship a version safely. What remains is the skill you will use most: working out why a pod is not running, with a method rather than a guess. That is the next lesson, and it collects the symptoms every previous lesson has mentioned into one order of operations.

Before that, run a rollout to an image tag that does not exist and watch what happens: the new pod fails to pull, the rollout stalls, and your application keeps serving from the old pods. Then undo it. Rehearsing a failed release while nothing is at stake is the whole point.